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Overview 


This report documents the impacts three years after random assignment for the Pathways to 
Healthcare program, operated by Pima Community College and Pima County One Stop in 
Tucson, Arizona. The program aimed to help low-income, low-skilled adults access and 
complete occupational training that could lead to increased employment and higher earnings. 
Pathways to Healthcare consisted of five elements: 


(1) Mapping 16 existing healthcare occupational training programs into five pathways, each 
incorporating a ladder that enabled students to obtain stackable credentials; 


(2) Proactive advising such as career counseling; 
(3) Scholarships for tuition and books; 


(4) Two compressed basic skills programs that in one semester remediated students whose low 
skills prevented them from enrolling directly in training; and 


(5) Job search assistance. 


Pathways to Healthcare is part of the Pathways for Advancing Careers and Education 
(PACE) study. Funded by the Administration for Children and Families (ACF) within the U.S. 
Department of Health and Human Services, PACE is a multi-site experimental evaluation of 
nine programs aimed at helping low-income adults to access career pathways. The Career 
Pathways Intermediate Outcomes (CPIO) Study extends the follow-up period to three years for 
programs in the PACE Study. Future reports produced by the Career Pathways Long-Term 
Outcomes Study will extend the follow-up period further. 


Purpose 


Pathways to Healthcare was an effort to address the increased demand in the local labor 
market for workers in the healthcare sector and provide improved labor market opportunities for 
low-income, low-skilled adults. It operated between 2010 and 2016 with funding from the ACF 
Health Profession Opportunity Grants (HPOG) Program. Pathways to Healthcare was organized 
as a pathway that trained students for entry-level occupations, such as Certified Nursing 
Assistant and Medical Office Clerk, as well as for higher-level positions, such as Licensed 
Practical Nurse and Health Information Technology-related jobs. It incorporated a range of 
financial, academic, employment, and personal supports and services. 


This research was undertaken to evaluate whether Pathways to Healthcare was successful in 
providing training to low-income, low-skilled adults and whether the program’s efforts led to 
impacts on credentials, earnings, healthcare employment, and other life outcomes. 
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Research Questions 
Three years after random assignment, what were the effects of Pathways to Healthcare on: 


e educational outcomes? 
e entry into career-track employment and higher earnings? 
e individual and family well-being, including income and other life outcomes? 


Key Findings 
Analyses in this report indicate that after three years, Pathways to Healthcare: 


m Increased the receipt of postsecondary credentials taking a year or more of 
college to earn—one of the two confirmatory outcomes in this report. 


The program increased the receipt of postsecondary credentials taking a year or more of 
college to earn from 11 percent to 18 percent, an impact of 7 percentage points. Credentials 
that typically require at least a year of college are associated with higher-level pathway courses, 
such as Licensed Practical Nurse and Medical Assistant. The program increased receipt of 
college credentials taking less than a year from 6 to 26 percent, a noticeably larger 20 
percentage point impact. Credentials associated with short-term training include Certified 
Nursing Assistant and Medical Office Clerk. 


@ Had no detectable impact on average quarterly earnings in follow-up quarters 12- 
13, the second of two confirmatory outcomes in this report. 


Treatment and control group members both earned about $4,000 per quarter in quarters 12 and 
13 after random assignment. There was no evidence of positive impacts on earnings through 
quarter 18. 


@ Had no detectable impact on employment overall as of three years after random 
assignment, but increased employment in the healthcare field. 


Slightly less than two thirds of both the treatment and control groups reported being employed 
as of the follow-up survey. About 40 percent of the treatment group self-reported employment in 
the healthcare field, an increase of 5 percentage points over the control group. There were no 
detectable impacts on characteristics of job quality, such as jobs that pay at least $14 per hour, 
require “at least mid-level skills,” or offer health insurance. 


Methods 


The Pathways to Healthcare evaluation used an experimental design in which program 
applicants were assigned at random to a treatment group that could access the program ora 
control group that could not, then compared their average outcomes. From February 2012 to 
February 2014, more than 1,200 applicants were randomly assigned. The impact study used 
data from a follow-up survey conducted three years after random assignment, administrative 
records from Pima Community College, earnings records from the National Directory of New 
Hires, and college enrollment data from the National Student Clearinghouse. The study 
measured impacts on training, employment, and earnings outcomes approximately three years 
after random assignment for all measures and up to five years for select earnings and education 
measures with available administrative data. 
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Executive Summary 


Pima Community College (PCC) and Pima County One-Stop (PCOS) in Tucson, Arizona, 
implemented the Pathways to Healthcare Program to help low-income, low-skilled adults 
access and complete healthcare occupational training that could lead to increased healthcare 
employment and higher earnings. In doing so, PCC also aimed to address expected labor 
shortages in healthcare occupations in the Tucson area. 


Abt Associates is evaluating Pathways to Healthcare as part of the Pathways for Advancing 
Careers and Education (PACE) study. Funded by the Administration for Children and Families 
(ACF) within the U.S. Department of Health and Human Services, PACE is a multi-site 
experimental evaluation of nine programs aimed at helping low-income adults to access career 
pathways. This report, which is part of the Career Pathways Intermediate Outcomes study, 
summarizes Pathways to Healthcare’s impacts on educational attainment, earnings and 
employment, and other life outcomes over a three- to five-year follow-up period.* It extends 
analyses from an earlier report that covered implementation and short-term impacts (18 months 
after randomization) on education and employment-related outcomes.” Future reports produced 
by the Career Pathways Long-Term Outcomes Study will extend the follow-up period further. 


The Pathways to Healthcare Evaluation 


Pathways to Healthcare aimed to engage low-income adults in college and facilitate their 
academic and career progress by augmenting the Pima Community College’s existing training 
programs with support and employment services. It operated between 2010 and 2016 with 
funding from the ACF Health Profession Opportunity Grants (HPOG) Program.° 


Pathways to Healthcare mapped 16 existing healthcare occupational training programs into five 
pathways and aspired to guide participants along them: Medical Office, Nursing, Medical and 
Physician Support, Emergency Medicine, and Other. Each pathway incorporated multiple steps, 
and each step (except the last) was associated with a higher-level credential. The program also 
included proactive academic and non-academic advising; scholarships for tuition and books; 
two compressed basic skills programs that in one semester remediated students whose low 
skills prevented them from enrolling directly in training; and job search assistance. 


All outcomes are measured at least three years after random assignment. For some outcomes in the 
educational attainment and earnings and employment domains, administrative records are available 
for up to five years after random assignment. 


That implementation and early impact report is available at 
https://www.acf.hhs.gov/opre/resource/pima-community-college-pathways-to-healthcare-program- 


implementation-early-impact-report. 
ACF is separately evaluating the HPOG Program through the HPOG 1.0 Impact Study. Pathways to 
Healthcare is included in the pooled analyses of that study. For more information see 


httos://www.acf.hhs.gov/opre/research/project/health-profession-opportunity-grants-hpog-impact- 


studies. 
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Abt Associates used an experimental evaluation design to estimate the impact of access to 
Pathways to Healthcare on participants’ postsecondary training, earnings and employment, and 
other life outcomes.’ A total of 1,217 individuals agreed to participate in the study—609 were 
randomly assigned to the treatment group and 608 to the control group. The analysis estimates 
impacts for each outcome by calculating the difference between average values in the treatment 
and control groups. The experiment was designed to capture the effects of the program overall 
rather than the separate contributions of its components. ° 


The short-term report indicated that PCC implemented as designed the program’s key 
components, including basic skills education, occupational training pathways, and advising. In 
general, treatment group members received more supportive services than control group 
members did, though the proportion of each group receiving services was low. For example, 23 
percent of treatment group members and 14 percent of control group members reported 
receiving employment services. Though the 9 percentage point difference is statistically 
significant, it remains that the majority of treatment group members did not receive employment 
services. 


As of 18 months after random assignment, Pathways to Healthcare increased hours of 
occupational training—the short-term confirmatory outcome pre-selected to assess whether the 
program was on track to meet its longer-term education and earnings goals. The program also 
increased college credentials earned. 


Key Findings from the Current Report 


This report begins by describing participation of the treatment group in Pathways to Healthcare 
training since the short-term report. Then the focus shifts to the impact of Pathways to 
Healthcare on postsecondary training, earnings and employment, and other life outcomes over 
a three- to five-year follow-up period. 


Participation in Training 


The Pathways to Healthcare program first aimed to increase participation in and completion of 
an initial step on a career pathway; next it advised participants to work in a healthcare job 
related to the credential earned and later to return and enroll in a subsequent step on the 
pathway. 


Such a design ensures that any estimated impacts can be attributed to program access rather than to 
unmeasured differences between eligible study sample members with access (the treatment group) 
and without access (the control group). 


Designers of Pathways to Healthcare deliberately included a package of multiple strategies (e.g., 
assessment, instruction, Supports, and employment connections) that they hypothesized were 
needed to produce desired impacts. Thus, the evaluation focuses on whether the program as a 
whole, when implemented in real-world conditions, produced an impact. 
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m The treatment group’s participation in Pathways to Healthcare occupational 
training and their progress along its pathways grew over the three years of follow- 
up. 

As of three years after random assignment, 56 percent of the treatment group had enrolled in 
occupational training, an increase from 48 percent in the first 18 months. Overall, 19 percent of 
treatment group members earned a credential at a level higher than where they began their 
training, indicating that they took a next step on their career pathway. 


The Pathways to Healthcare program ended in 2016, but among the options available to 
treatment group members to continue their progress was to enroll in further training through 
PCC’s second-round HPOG grant, the Health Career Opportunities with Personalized 
Educational Supports (HOPES), which is expected to end in 2020.° Or they could enroll in 
other programs at PCC or elsewhere in the community. According to HOPES program records, 
84 treatment group members, or nearly 14 percent of the Pathways to Healthcare treatment 
group, received training as part of HOPES.’ 


Impacts on Postsecondary Training 


Pathways to Healthcare continued to have impacts on credential receipt after the impacts 
observed in the 18-month report; that is, the program’s impact on receipt of college credentials 
taking less than a year to earn (“short-duration” credentials) persisted. At three years after 
randomization, there was an impact of the program on credentials taking a year or more of 
college to earn (“long-duration” credentials)—pre-selected as the confirmatory outcome at this 
later follow-up point. The program also increased the number of college credits earned and 
overall enrollment in college at three years after random assignment. 


m The treatment group earned more college credentials that take a year or more to 
earn—the confirmatory educational outcome in this three-year report—than did 
the control group. 


Credentials that typically require at least a year of college are associated with higher-level 
pathway courses, such as Licensed Practical Nurse and Medical Assistant. Pathways to 
Healthcare increased the receipt of these types of credentials from 11 percent in the control 
group to 18 percent in the treatment group, an impact of 7 percentage points (Exhibit ES-1). 


° The second HPOG grant is expected to end in September 2020. As of a fall 2017 site visit, PCC 
expected to sustain the program after the grant ends, although state and college budget cuts could 
affect these plans. 


If Pathways to Healthcare participants enrolled in or completed credentials through HOPES during 
the three-year follow-up period, that enrollment/attainment was counted as part of the three-year 
outcomes analyzed in this report. See Chapter 3. 
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Exhibit ES-1: Impacts on Postsecondary Training Outcomes, Three Years after Randomization 


50 


N Ww & 
oO oO oO 


Percentage (%) 


anh 
oO 


0 
CONFIRMATORY: Received any college Received healthcare Received exam-based 
Received 1+ year credential credential from a certification or license 
college credential college 


Control Group {J Treatment Group 


Source: PCC records: Received 1+ year college credential, Received any college credential, and Received healthcare credential from a 
college. Blend of PACE 18-month and three-year follow-up surveys: Received exam-based certification or license. 


Note: Sample size for Received 1+ year college credential, Received any college credential, and Received healthcare credential from a 
college is 609 in the treatment group and 608 in the control group. Sample size for Received exam-based certification or license is 530 in the 
treatment group and 510 in the control group. Statistical significance is based on one-tailed tests. Receipt of exam-based certification or 
license is based on responses from both the 18-month and three-year follow-up surveys. 


Statistical significance levels based on differences between research groups: *** 1 percent level; ** 5 percent level; * 10 percent level. 


m Pathways to Healthcare had larger impacts on receipt of any college credentials 
defined broadly than it did on long-duration college outcomes specifically. 


Exhibit ES-1 also reports impacts for secondary outcomes in the postsecondary training 
domain, including received any college credential and received a healthcare credential from a 
college. As shown, Pathways to Healthcare’s 24 percentage point impact for any credential 
(which includes both short-duration and long-duration ones) is more than three times as large as 
the impact for long-duration credentials only (7 percentage points). The impact is an increase of 
140 percent above the control group mean (16 percent). The program also had a 24 percentage 
point impact on receipt of a healthcare credential from a college. There was no detectable 
difference in receipt of exam-based certifications or licenses. 


m Pathways to Healthcare increased college enrollment and credits. 


The program increased full-time-equivalent months enrolled in college from a control group 
mean of five months by 0.8 months according to PCC administrative records and by 2.8 months 
according to the three-year follow-up survey. The program also increased the average number 
of college credits earned by 2.3 credits (Neither shown). 
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Impacts on Earnings, Employment, and Other Life Outcomes 


Pathways to Healthcare’s impact on credentials generated little detectable impact on 
employment, earnings, or other life outcomes over the nearly five-year follow-up period 
observed. 


™ Pathways to Healthcare had no detectable impact on average quarterly earnings in 
follow-up quarters 12-13—the confirmatory earnings outcome in this three-year 
report. 


Exhibit ES-2 shows that treatment and control group members both earned about $4,000 per 
quarter in follow-up quarters 12 and 13. At that time, the difference in average quarterly 
earnings was positive but small (+$17) and not statistically significant. As is true in all 
evaluations of job training programs, the impact was estimated with uncertainty. A plausible 
range of estimates taking into account this uncertainty would be -$322 to +$356.° Most of this 
range is either negative or not sufficiently larger than zero to be meaningful.° The exhibit also 
shows no evidence of positive earnings impacts through quarter 18. Thus, the treatment group’s 
additional training has not yet produced increased earnings. 


m Pathways to Healthcare had no detectable effect on employment overall as of 
three years after random assignment, but it did increase self-reported employment 
in the healthcare field. 


Slightly less than two thirds of both the treatment and control groups reported employment at 
the time of the three-year follow-up survey. Nearly four of 10 treatment group members self- 
reported employment in the healthcare field, an impact of 5 percentage points relative to the 
control group. 


There were no detectable impacts on other characteristics of employment. This included jobs 
that pay at least $14 per hour, jobs that require “at least mid-level skills,” hours worked per 
week, type of shift (regular or varying), availability of health insurance, or a supportive working 
environment.”° 


These values are the endpoints for a 90 percent confidence interval for average earnings in quarters 
12 to 13. 


The upper end of this plausible range is not large relative to results from some recent studies. For 
instance, about two years after random assignment, impact for those assigned to the Wisconsin 
Regional Training Partnership (WRTP) in the Sectoral Employment Impact Study (SEIS) was $782 
per quarter (Maguire et al. 2010) and for Per Scholas (one provider in the WorkAdvance 
Demonstration) was $937 per quarter (Hendra et al. 2016). 


*° We tested whether the program could generate a moderate initial boost in wages with the expectation 


for further impact over time (a key assumption in the theory of change). We defined these jobs using 
the 60th percentile of the control group wage distribution, which corresponded to $14 per hour. 


Abt Associates Executive Summary | pg. X 


Pathways to Healthcare Program: Three-Year Impact Report 


Exhibit ES-2: Impact on Average Earnings in Successive Follow-up Quarters, Three Years after 
Randomization 
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Average Quarterly Earnings 
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0 Q-2 Q-1 QO Qi Q2 Q3 Q4 QS Q6 Q7 Q8B QY Q10 Q11 Q12 Q13 Q14 Q15 Q16 Q17 Q18 
Quarter Since Random Assignment 
Source: National Directory of New Hires. 
Note: Earnings estimates within each quarter are exploratory outcomes and statistical significance is based on two-tailed tests. Sample size 


is 609 in the treatment group and 608 in the control group. 
Statistical significance levels based on differences between research groups: *** 1 percent level; ** 5 percent level; * 10 percent level. 


m Findings were mixed for career knowledge, availability of career supports, and 
psycho-social skills. 


Improvements to these outcomes are hypothesized to boost postsecondary educational 
attainment and career progress. As well, improvements in educational attainment and career 
progress are expected to positively affect these outcomes. The treatment group reported 
greater access to career supports, but no more confidence in their career knowledge or higher 
levels of psycho-social skills. 


m Pathways to Healthcare had no detectable impact on most measures of family 
economic well-being. 


The Pathways to Healthcare theory of change implies that outcomes related to family economic 
well-being will improve as a result of increases in education and training leading to more 
favorable earnings and employment outcomes. However, the program did not have any 
detectable impact in this area. The results do not show a statistically significant difference in 
health insurance coverage between the treatment and control groups; roughly 90 percent of 
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both groups reported having coverage. Nor did the program have a detected impact on receipt 
of means-tested public benefits, participant student debt, or any signs of financial distress. ** 


Possible Explanations 


To summarize the main results over the three- to five-year follow-up period covered in this 
report, Pathways to Healthcare increased educational credentials but had no impact on 
earnings. The report explores several possible explanations for the combination of positive 
impacts on education outcomes and no detectable impact on earnings: 


e A large share of treatment group members did not engage in the program. 


Forty-four (44) percent of the treatment group did not engage in any occupational training within 
three years after random assignment. This is slightly lower than in the 18-month follow-up 
period, when 52 percent did not participate in occupational training, because some treatment 
group members who had not engaged in the short-term had engaged within three years. If 
occupational training is required in order to observe an impact on earnings, then the large 
minority of treatment group members who did not engage reduces average impacts for the 
study sample as a whole. 


e The program mostly increased short-duration credentials with modest potential to 
lift wages. 


Pathways to Healthcare had large impacts on short-duration credentials (20 percentage points). 
Short-duration credentials are typically associated with initial pathway steps, and wages for 
occupations in those steps do not differ substantially from wages that can be earned outside of 
the healthcare field (e.g., personal care and service, food preparation or serving). Thus, a short- 
duration credential may position the worker on the initial step of a career ladder, but absent 
follow-on training, the credential would not necessarily generate an impact on earnings. 


e Impacts on long-duration credentials are too small to generate earnings impacts. 


Average hourly wages should grow when participants move to occupations associated with 
higher-level credentials. The relatively small impact of the Pathways to Healthcare program on 
receipt of long-duration credentials implies impacts on earnings may not be large enough for the 
study to detect. 


‘1 Unsecured debt is debt other than student debt and secured debt (mortgages and title loans); spousal 


debt is included. Signs of financial distress is a flag for utility disconnects, delayed health/dental care, 
hunger, or trouble paying bills or making ends meet. 
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Looking Ahead 


A future report will document the impact of Pathways to Healthcare on educational progress and 
earnings approximately six years after random assignment. In that report, detectable impacts on 
earnings would be expected only if there are changes along the dimensions discussed in the 
prior section; that is, if more treatment group members engage in training associated with 
longer-duration credentials and shift into higher-paying jobs. That six-year follow-up research 
will explore whether impacts on long-duration credentials grow and impacts on earnings 
emerge. 
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The federal Bureau of Labor Statistics (BLS) projects that healthcare occupations will add more 
jobs through 2026 than any other occupational group, largely due to an aging population. The 
healthcare sector’s projected growth rate, 18 percent, is much faster than any other 
occupational group.” 


Community colleges and other training providers are offering a range of training programs 
tailored to this increased demand in the healthcare sector. Many of these programs are 
organized as pathways that train students for entry-level occupations, such as Certified Nursing 
Assistant and Medical Office Clerk, as well as higher-level positions, such as Licensed Practical 
Nurse and jobs in Health Information Technology. These programs range in length from one to 
24 months, and result in a variety of credentials ranging from occupational certificates to 
associate degrees. Moreover, programs aim to enroll a variety of students—in particular 
nontraditional students—who are older, are likely to be combining work and school, and have 
children. To do so, they offer accelerated coursework, flexible formats (e.g., online, at different 
locations in the community), and incorporate basic skills remediation for those who have been 
out of school for a period of time. 


In recent years, federal initiatives have aimed to help low-income adults, including Temporary 
Assistance for Needy Families (TANF) recipients, access training in order to attain skills needed 
to become self-sufficient.** The Administration for Children and Families (ACF) within the U.S. 
Department of Health and Human Services funded one such initiative, the Health Profession 
Opportunity Grants (HPOG) Program. In 2010, ACF awarded a first round of HPOG grants 
(HPOG 1.0) to programs that provided TANF recipients and other eligible low-income 
individuals with the opportunity to obtain education and training for occupations in the 
healthcare field that were expected to either experience labor shortages or be in high demand.“ 


12 See https://www.bls.gov/ooh/healthcare/home.htm. 


3 The TANF program, which is time limited, assists families with children in providing for the family's 


basic needs. The Federal government provides block grants to States to run the TANF program. 
States have broad flexibility to carry out their programs. States determine the design of the program, 
the type and amount of assistance payments, the range of other services to be provided, and the 
rules for determining who is eligible for benefits. 


The HPOG Program was authorized by the Affordable Care Act (ACA), Public Law 111-148, 124 Stat. 
119, March 23, 2010, sect. 5507(a), “Demonstration Projects to Provide Low-Income Individuals with 
Opportunities for Education, Training, and Career Advancement to Address Health Professions 
Workforce Needs,” adding sect. 2008(a) to the Social Security Act, 42 U.S.C. 1397g(a), and extended 
by the Bipartisan Budget Act of 2018, Pub. L. 115-123, through fiscal year 2019. ACF awarded a 
second round of grants in 2015. 
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Pima Community College (PCC) in 
Tucson, Arizona, received an HPOG 
1.0 grant to implement the Pathways 
to Healthcare Program. The 
program provided financial support 
for training, along with case 
management, supportive services, 
and employment services, to 
encourage low-income adults to train 
for healthcare jobs. It operated 
through a partnership between PCC 
and Pima County One-Stop (PCOS) 
from 2010 to 2016.*° 


Abt Associates is evaluating 
Pathways to Healthcare as part of 
the Pathways for Advancing 
Careers and Education (PACE) 
study. Funded by ACF, PACE is 
studying nine programs aimed at 
helping low-income adults to access 
career pathways (see Programs in 
PACE box). This study, the Career 
Pathways Intermediate Outcomes 
study, extends the follow-up period to 
three years for programs in the 
PACE study. Future reports 
produced by the Career Pathways 


Programs in PACE 


e Bridge to Employment in the Healthcare Industry, 
San Diego Workforce Partnership, San Diego County, 
CA* 


e Carreras en Salud, Instituto del Progreso Latino, 
Chicago, IL” 


e Health Careers for All, Workforce Development 
Council of Seattle-King County, Seattle, WA* 


e Integrated Basic Education and Skills Training 
(I-BEST) program at three colleges (Bellingham 
Technical College, Everett Community College, and 
Whatcom Community College), Washington State 


e Pathways to Healthcare, Pima Community College, 
Tucson, AZ* 


e Patient Care Pathway Program, Madison College, 
Madison, WI 


e Valley Initiative for Development and 
Advancement (VIDA), Lower Rio Grande Valley, TX 


e Workforce Training Academy Connect, Des Moines 
Area Community College, Des Moines, IA 


e Year Up, Atlanta, Bay Area, Boston, Chicago, 
National Capital Region, New York City, Providence, 
and Greater Seattle 


*Programs funded through the Health Profession Opportunity 
Grants (HPOG) Program. 


“Program partially HPOG funded. 


Long-term Outcomes Study will extend the follow-up period further. 


All nine programs include some features of the overarching career pathways framework (Fein 
2012). This framework posits that postsecondary education and training should be organized as 
a series of steps leading to successively higher credentials and employment opportunities in 
growing occupations. To effectively engage, retain, and facilitate learning of a diverse 
population, career pathways programs integrate four program components: 


(1) Academic and non-academic assessment to identify student needs and factors that may 
facilitate or hinder academic success, so advisors can make appropriate placements and 


referrals; 
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Grants were five years in duration. Grantees could request a six-month no-cost extension. PCC 


received a six-month extension and provided services through the Pathways to Healthcare program 


through March 2016. 
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(2) Innovative basic skills and occupational skills instruction to make education and 
training more manageable for nontraditional students who are likely to be balancing school 
and work (e.g., accelerated courses) and who may have low levels of basic skills (e.g., 
contextualization); 


(3) Academic and non-academic supports (e.g., academic advising, tutoring, financial 
support, and referrals to support services) to help students succeed in their current 
academic step and to proceed to and complete subsequent steps; and 


(4) Strategies to connect participants and employers during the program, such as 
internships, or post program, such as employment workshops. 


Because the nine programs vary in their target populations, mix of components, and 
occupational fields, PACE is evaluating each program separately.*® This report documents the 
impact of Pathways to Healthcare on postsecondary training, earnings and employment, and 
other life outcomes of students approximately three years after they agreed to participate in an 
evaluation of the program. An earlier report shared findings on implementation and short-term 
(18-month) impacts on education, employment, and related outcomes (Gardiner et al. 2017). 


The remainder of this chapter describes Pathways to Healthcare key components and context 
(Section 1.1). It then summarizes findings from the 18-month report (Section 1.2). Finally, it 
provides a roadmap to the remainder of the report (Section 1.3). 


1.1 The Pathways to Healthcare Program 


ACF awarded PCC of Tucson, Arizona, a five-year, $18-million HPOG grant to launch and 
operate the Pathways to Healthcare program. The program ended in 2016.*’ 


The major Pathways to Healthcare program components were: 


e Mapping 16 of PCC’s occupational training programs into five pathways and 
identifying a “ladder” of programs within each pathway (Levels 1 through 3). This 
structure clarified for students how completion of one credential can lead to a next- 
higher credential (and a higher-paying job) (See Exhibit 1-1). Moreover, Level 1 
programs included all prerequisites for Level 2, and Level 2 programs included 
prerequisites for Level 3. PCC tied learning outcomes to expected outcomes in the 
industry, with each level indicating greater mastery. This structure made the sequencing 
of programs and credentials within each pathway clear. The 16 occupational training 
programs varied along several dimensions: in length (from one to 24 months), in whether 


© PACE-related documents, including profiles and implementation and early impact reports for each 


program, can be found at www.acf.hhs.gov/opre/research/project/pathways-for-advancing-careers- 
and-education and www.career-pathways.org. 


PCC began implementing a modified version of the Pathways to Healthcare program, Health Career 
Opportunities with Personalized Educational Supports (HOPES), with funding from a second-round 
HPOG grant (HPOG 2.0) awarded in 2015 and implemented beginning in 2016. 
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completers obtained credits or noncredit “clock hours,” and in whether completion 
resulted in a degree or a certificate. 


e More-intensive and proactive academic and non-academic advising to identify and 
resolve barriers to program participation, aid participants in selecting a training program, 
help them persist in and complete the chosen program, and subsequently enter 
employment. Participants had three advisors: one at PCOS to identify and address 
barriers to participation and two at PCC. Students worked with one PCC advisor when 
they were preparing for occupational training (e.g., filling out required paperwork, 
scheduling classes, attending a College Readiness class or lab) and a different PCC 
advisor after their program began. 


e Scholarships for tuition, books, and other program supplies, to reduce the cost of 
obtaining a certificate or degree. 


e Two compressed and contextualized College Readiness basic skills “bridge” 
programs for participants who did not test high enough on college assessments to enter 
their occupational training program directly.*® The 10-week course and the self-paced 
open lab aimed to help students quickly remediate their basic skills so that they could 
retake and pass the assessments.’° 


e Employment supports, including resume preparation, coaching for interviews and other 
forms of job search assistance to help program completers locate employment, most of 
which staff developed in the later years of the Pathways to Healthcare program. 


Applicants deemed eligible for Pathways to Healthcare first took the TABE to determine whether they 


needed to start at the College Readiness level, and then took the ACT Compass to determine their 
starting level within a pathway. See Section 1.1.1 for details on eligibility requirements. 


= Participants could select either College Readiness bridge format; selection was generally a function 


of schedules. Participants who worked during the day and could not commit to a structured class 
tended to enroll in the lab. 
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Exhibit 1-1: Pathways Overview, Pathways to Healthcare 


Medical Office Pathway 
*Health Information 


Technology 
Medical Office Pathway ; 
*Medical Records Nursing Pathway 
Technician ¢Licensed Practical Nurse 
ee one ey i Medical and Physician 
a UNG | «Medical Office Clerk / Nursing Pathway é ysicia 
BRIDGES Specialist *Patient Care Technician | | Support Pathway 
*Clinical Research 
College Readiness | | Nursing Pathway Necetiectgeber silt Coordinator 
Class *Certified Nursing Assistant upport Pathway he 
(10 weeks) 2 «Medical Assistant surehaae Medicine 
or Medical and Physician = pera ae 
College Readiness | | Support Pathway ae Medicine —Paramedicine 
Open Lab *Phlebotom atnway 
(«K< 6 months) 4 *EMT—Basic Other Pathway 
Emergency Medicine ¢Pharmacy Technician 
Pathway Other Pathway ) 
None *Surgical Technician 


Other Pathway 

*Community Health Advisor 
«Direct Care Professional 
¢Behavioral Health Specialist 


As noted above, the pathways were designed so that each level completed was associated with 
a higher-paying position. Hourly wages for Level 1 occupations were not substantially different 
from entry-level wages outside of the healthcare field. For instance, according to the BLS, the 
average wage in the Tucson Metropolitan Statistical Area (MSA) in May 2018 was $14.36 per 
hour for a Certified Nursing Assistant; $14.97 for a health support worker; and $16.84 for a 
Phlebotomist.*° These wages are only slightly higher than other initial career ladder steps 
including personal care and service ($13.39 per hour) and food preparation and serving jobs 
($13.15 per hour). Furthermore, these are average wage rates; at entry level the wages are 
likely lower than reported here. 


As participants progressed along pathways, they could expect gradual wage growth. Average 
wages for Level 2 occupations such as Medical Assistant and Surgical Technician were $15.04 
and $22.74 per hour, respectively. Average wages for Level 3 occupations such as Pharmacy 
Technician and Licensed Practical Nurse were $17.47 and $24.26, respectively. 


Each occupational training in Pathways to Healthcare involved upfront activities such as the 
training-specific “passport,” which was a checklist of steps students must have completed prior 
to starting a training program. Program participants also worked with their advisors to create a 
Training Plan, which listed preparatory activities required to enroll in a particular training 
program. However, depending on when a participant entered Pathways to Healthcare, he or she 


20 


See https://www.bls.gov/oes/current/oes 46060.htm. 
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might have had to wait for a training class to start. On average, the length of time from program 
enrollment to the start of the first training class was four months. Toward the end of the grant 
period, PCC implemented a bi-monthly group orientation to engage participants and remind 
them of the steps required prior to entering training. 


PCC designed the program so that students could take multiple steps on a pathway. As noted, 
each pathway included a natural sequencing of training programs and associated credentials 
that recognized student progress. With the exception of one program pairing (Certified Nursing 
Assistant to Patient Care Technician), completers had to work for at least six months before 
returning for the next level of training to ensure that the occupation was a good fit, and then take 
the initiative to re-enroll in the next pathway course of study. The time-limited funding through 
the HPOG 1.0 grant may have curtailed some students’ ability to return and earn a higher-level 
credential.** For example, a treatment group member who enrolled in a training program toward 
the end of the Pathways to Healthcare grant period would not have had time to complete 
training, obtain work experience, and return for a second training before the HPOG 1.0 funding 
expired in March 2016.” 


1.1.1 Eligibility and Enrollment 


For the PACE evaluation of Pathways to Healthcare, PCC staff screened applicants for eligibility 
and then randomly assigned eligible applicants to a treatment or a control group. To be eligible 
for the program, applicants had to: 


e Reside in Pima County; 


e Have income at or below 70 percent of the Lower Living Standard Income Level;”° 


e Be eligible to work in the United States; and 


e Have an interest in a healthcare-related career. 


Applicants deemed eligible who consented to be in the study completed two study forms (the 
Basic Information Form and the Self-Administered Questionnaire). Random assignment 
governed program entry. Those study participants randomly assigned to the treatment group 


21 HPOG 2.0 grants were awarded in September 2015. As a result, HROG 1.0 grantees who were 


awarded HPOG 2.0 grants (Such as PCC) did not know they could continue supporting students until 
the end of the HPOG 1.0 grant period. 


To address this issue, per ACF guidance, PACE study participants could enroll in additional training 
through HOPES, PCC’s second-round HPOG-funded program. According to the research team 
evaluating HPOG 2.0, PCC staff conducted outreach to HPOG 1.0 treatment group members to alert 
them to its new grant and the opportunity to obtain additional training without having to go through the 
new evaluation’s random assignment. 


22 


3 Lower Living Standard Income Level is a measure, determined by the U.S. Department of Labor, to 


establish “low-income” status. In 2013, for a family of four in Pima County, 70 percent of the Lower 
Living Standard Income Level was $27,724 annually. This is about 16 percent higher than the poverty 
guideline the same year for a family of four ($23,850). See https://aspe.hhs.gov/2014-poverty- 


quidelines. 
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could access Pathways to Healthcare; those assigned to the control group could not.”* Between 
February 2012 and January 2014, PCC staff randomly assigned 1,217 study participants: 609 to 
the treatment group and 608 to the control group. 


Although the control group could not enroll in Pathways to Healthcare, they could enroll in other 
services and programs in the community. In practice, control group members had access to 
much, but not all, of the services available to treatment group members. With the exception of 
One evening Nursing Assistant course taught by an HPOG-funded instructor, control group 
members could enroll in PCC’s 16 occupational programs mapped to Pathways to Healthcare, 
but they would need to seek financial support on their own. Thus, the key treatment-control 
group differences were the availability of: 


e The College Readiness bridges for those who needed them; 
e Financial assistance in the form of scholarships; 


e Academic advising to help navigate the enrollment process and to provide academic 
supports; 


e Non-academic advising to identify and address challenges to enrollment and persistence 
in the program; and 


e Dedicated employment services. 


1.1.2 Characteristics of the Study Sample 


Exhibit 1-2 shows the study sample’s characteristics at baseline, both overall and for the 
treatment and control groups separately. The p-values in the right-most column indicate that the 
evaluation’s random assignment procedure produced treatment and control groups with no 
significant differences in these characteristics. 


Exhibit 1-2 also shows that the Pathways to Healthcare study sample reflects the program's 
eligibility criteria closely. At application, sample members were low income; approximately half 
had annual household incomes of less than $15,000, and about 85 percent had incomes less 
than $30,000. About two thirds received benefits from the Supplemental Nutrition Assistance 
Program (SNAP) or the Special Supplemental Nutrition Program for Women, Infants, and 
Children (WIC). TANF recipients, a target population for the HPOG Program generally, 
accounted for about 8 percent of study participants. 


About three fifths reported experiencing signs of financial hardship in the past year.*° Most 
study participants were not working at the time of random assignment, although the majority 


= Eligible applicants who did not consent to be in the study did not fill out the study forms and were not 


able to enroll in Pathways to Healthcare. 


2° Financial hardship is defined as ever missed rent/mortgage payment in prior 12 months or reported 


generally not having enough money left at the end of the month to make ends meet over the last 12 
months. 
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expected to start working in the following months, perhaps suggesting a need or desire to 
combine work and education and training. 


Exhibit 1-2: Selected Characteristics of the Pathways to Healthcare Sample at Baseline 


All Treatment Control 
Characteristic Participants Group Group p-Value 
Age (%) 320 
20 or under 8.4 6.9 9.8 
21 to 24 13.0 13.3 12.8 
25 to 34 31.7 32.2 31.3 
35 or older 46.9 47.6 46.2 
Sex (%) .929 
Female 82.7 82.8 82.7 
Male 17.3 17.2 17.4 
Race/Ethnicity (%) 631 
Hispanic, any race 55.8 56.4 55.2 
Black, non-Hispanic 11.5 12.3 10.7 
White, non-Hispanic 26.6 26.6 26.6 
Other, non-Hispanic 8.0 7.0 9.1 
Current Education (%) 179 
Less than a high school diploma 8.4 7.9 8.8 
High school diploma or equivalent 34.5 35.4 33.6 
Less than 1 year of college 16.4 16.9 15.9 
1 or more years of college 26.3 26.5 26.0 
Associate degree or higher 14.5 13.3 15.7 
Income (%) 618 
Less than $15,000 48.9 50.1 47.6 
$15,000 to $29,999 36.2 35.9 36.6 
$30,000 or more 14.9 14.1 15.8 
Mean ($) $17,236 $16,817 $17,653 294 
Public Assistance / Hardship in Past 12 Months (%) 
Received WIC or SNAP 68.3 67.1 69.6 345 
Received public assistance or welfare 7.7 7.3 8.1 676 
Reported signs of financial hardship 59.4 61.6 57.3 .128 
Current Work Hours Per Week (%) 211 
0 65.6 66.9 64.3 
1to 19 6.8 5.6 8.0 
20 to 34 15.6 14.7 16.6 
35 or more 12.0 12.9 11.2 
Expected Work Hours Per Week in Next Few Months (%) 408 
0 30.4 30.4 30.5 
1to 19 5.8 48 6.8 
20 to 34 37.4 37.0 37.7 
35 or more 26.5 27.8 25.0 
Sample sizes 1,217 609 608 


Key: SNAP = Supplemental Nutrition Assistance Program. WIC = Special Supplemental Nutrition Program for Women, Infants, and Children. 
Source: PACE Basic Information Form. 
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Note: There are no significant differences at the p = .10 level. Some percentages for characteristics do not add up to 100% due to rounding. 
Public Assistance/Hardship in Past 12 Months does not add to 100% because the categories are neither mutually exclusive nor exhaustive. 
See Appendix A for more details on baseline characteristics. 


In terms of demographics, study participants were older than traditional college students. About 
80 percent were age 25 and older, and almost half were age 35 or older. In terms of education, 
about 60 percent had less than a high school education, a high school diploma or equivalent, or 
less than a year of college. The great majority of study participants were female. More than half 
identified as Hispanic. 


1.1.3. Local Context 


Pima County, located in southern Arizona on the border with Mexico, is urban and ethnically 
diverse. At the time of the study, the county had more than one million residents; Tucson, with a 
population of 527,972, was the largest city.*° The median household income in Pima County 
was $46,233, lower than for Arizona ($49,928) and the United States ($53,657).7” 


Like the national economy, the local economy improved over the study period. Between March 
2013 and March 2015, the Pima County unemployment rate decreased from 6.6 percent to 5.3 
percent. By March 2018 (the end of the current report’s observation period), it had fallen further 
to 4.4 percent. *° 


The types of healthcare jobs for which Pathways to Healthcare provided training were projected 
to grow locally. Occupational projections for the Tucson MSA indicate that between 2012 and 
2022, “healthcare practitioner and technical’”’ occupations will increase by 18 percent and 
“healthcare support’”° occupations by about 23 percent.** In its HPOG grant application, PCC 
also noted that every year thousands of older people retire to the Sun Belt (which stretches 
across the southern United States, including Tucson) and that Arizona, relative to other states, 
has a lower than average number of caregivers (2.3 per 1,000 residents compared with 3.5 
nationally). The aging population could easily “overwhelm” the healthcare system. °** 


6 U.S. Census Bureau, Vintage 2014 Population Estimates: Sub-county Population Places and MCDs. 


Populations for 2014. 


Pima County data from https://www.census.gov/quickfacts/table/PST045215/04019. Arizona data 
from https://www.census.gov/quickfacts/table/PST045215/04. U.S. data from 
https://www.census.gov/data/tables/time-series/demo/income-poverty/cps-hinc/hinc-01.html. 


See http://data.bls.gov. 


This occupational group includes physicians (e.g., internists, surgeons, and obstetricians), nurses 
(e.g., registered, midwives, nurse practitioners, Licensed Practical Nurses and Licensed Vocational 
Nurses), dietitians and nutritionists, pharmacists, physical therapists, medical and clinical laboratory 
technicians, medical records and health information technicians, among others. 


27 


28 


29 


0 This occupational group includes home health aides, nursing assistants, orderlies, occupational 


therapy assistants and aides, medical assistants, and phlebotomists, among others. 


= https://laborstats.az.gov/employment-forecasts 2012-2022 Tucson MSA Occupation Projections 


Tables & Graphs. 


= Application for Federal Assistance, Pathways to Healthcare Professions. 
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1.2 Earlier Findings on Pathways to Healthcare from PACE 


The earlier Pathways to Healthcare Implementation and Early Impact Report (Gardiner et al. 
2017) provides useful context for the current report. In its initial phase, the PACE evaluation 
assessed the Pathways to Healthcare program’s implementation and short-term (18-month) 
impacts. The PACE implementation study examined the design and operations of Pathways 
to Healthcare and analyzed participation patterns of treatment group members in training and 
other activities. Its short-term impact study measured the program’s effects on training, 
credentials, and self-reported employment and career progress. This section summarizes key 
findings from that report. 


1.2.1 Earlier Results from the Implementation Study 


This section summarizes program implementation and participants’ experiences in the program 
through 18 months after random assignment. 


@ More than 60 percent of treatment group members enrolled in education or 
training. 
Thirty-five (35) percent of treatment group members enrolled directly in occupational training 
and 26 percent enrolled in one of the College Readiness bridge options for students whose 
basic academic skills were too low to enter an occupational program directly. Half of bridge 
participants proceeded to occupational training. 


However, 38 percent of treatment group members did not engage in any education or training 
activities. Responses to the 18-month follow-up survey suggest that reasons for not enrolling 
included not enough time for work and for family and not enough financial aid. Also, as noted 
above, about four months on average elapsed between random assignment and the start of a 
training program, which may have discouraged some treatment group members from enrolling. 


@ Less than half of treatment group members enrolled in occupational training; 
those who did generally enrolled in Level 1 programs. 


As of the 18-month follow-up, 48 percent of treatment group members enrolled in occupational 
training. This proportion includes half of the College Readiness completers (13 percent of 
treatment group members) and those who enrolled directly in occupational training (35 percent 
of treatment group members). Nursing Assistant was the most commonly attended and 
completed program, accounting for 30 percent of all treatment group members and 63 percent 
of those who enrolled in occupational training. Another 10 percent attended other Level 1 
programs, including Medical Office Clerk (6 percent) and Phlebotomy (4 percent). Sixteen (16) 
percent participated in Patient Care Technician training, a short-term Level 2 program that 
enabled Nursing Assistants to work in hospitals and nursing homes. 


m The treatment group was more likely than the control group to participate in 
education and training. They were also more likely to use advising and 
employment services. 


The treatment group was more likely than the control group to participate in any education and 
training (60 percent of the treatment group versus 47 percent of the control group) and 
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healthcare-related training (37 percent versus 30 percent). Treatment group members were also 
more likely to participate in basic skills instruction than were control group members (18 percent 
versus 10 percent), with the College Readiness bridges likely accounting for the difference. 


In terms of support services, treatment group members were more likely than control group 
members to receive career counseling (28 percent versus 19 percent), help arranging supports 
(14 percent versus 7 percent), and job search assistance (23 percent versus 14 percent). 
Despite the treatment group engaging in these services to a greater degree than the control 
group, fewer than a quarter of treatment group members reported receiving each service. 


1.2.2 Earlier Results from the Impact Study 


The PACE research team designated a single educational measure—hours of occupational 
training—as the confirmatory indicator of the Pathways to Healthcare program’s success at 18 
months. The short-term analyses also assessed a variety of other education outcomes, as well 
as several employment-related outcomes believed to provide an early indication of expected 
longer-term educational, employment, and earnings impacts. 


m Pathways to Healthcare increased the total hours of college occupational training 
that students received (the confirmatory outcome at 18 months). 


Pathways to Healthcare increased the primary outcome of interest for the 18-month analysis 
period: hours of occupationally focused college training. The program had a 63-hour impact on 
total hours of training: 190 hours for the treatment group compared with 127 hours for the 
control group, a relative increase of 50 percent. Converting these estimates to weeks implies 
trainings were relatively short for both groups as of 18 months—the treatment group trained for 
nearly five weeks compared to three weeks for the control group, on average. The program also 
increased receipt of a credential from a college. 


= Pathways to Healthcare had mixed effects on measures of early career progress 
18 months after random assignment. 


The research team assessed the impact of the program on two dimensions of career progress. 
First, self-assessed progress toward career goals included measures of perceived career 
progress, confidence in career Knowledge, and access to career supports. The analyses found 
positive impacts for these measures, with the largest effect on perceived career progress. 
Second, the study considered three different employment outcomes. As of 18 months, there 
was no evidence of impact on employment in a job that paid at least $12 per hour, required at 
least mid-level skills, or was in a healthcare occupation. 


1.3. Guide to Rest of the Report 


This report has seven chapters. Chapter 2 summarizes the Pathways to Healthcare study 
design and analytic methods, including a discussion of the career pathways theory of change 
and its implied research questions. The chapter also documents how the study implemented 
random assignment and describes its principal data sources. 


Chapter 3 describes the flow of the treatment group through healthcare training in 
Pathways to Healthcare. The short-term report found that 38 percent of the treatment group 
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did not participate in any training by 18 months. This chapter begins by updating those 
estimates as of three years after random assignment. We also summarize the flow of 
participants through various components of Pathways to Healthcare. This descriptive analysis 
provides context for the estimates of impact on postsecondary training in the chapter that 
follows. 


Chapter 4 presents the three-year impact findings on postsecondary education and 
training. As noted above, at 18 months after random assignment, Pathways to Healthcare had 
increased the share of its participants who received training and who received a postsecondary 
credential taking less than a year of college to earn (“short-duration” credential). This chapter 
reports analyses of how those early gains in training and healthcare credentials evolved over 
time. With the extra follow-up time allowing study participants time to complete more training, 
we identified the impact on credentials requiring a year or more of college to earn (“long- 
duration” credentials, which include associate and higher degrees) as the most important 
outcome measure of program success in the education domain at three years. 


Chapter 5 presents the three-year impact findings on earnings and employment. The 
short-term impact study conducted a relatively limited analysis of impacts on labor market 
outcomes at 18 months because such impacts were expected to take longer to emerge. This 
three-year report provides more detail on impacts on labor market outcomes for a period when 
such impacts might plausibly emerge. We identified earnings as the most important outcome 
measure of program success in the earnings and employment domain. 


Chapter 6 presents the three-year impact findings on other life outcomes such as career 
knowledge, availability of career supports, psycho-social skills, family economic well-being, 
parental engagement, and child outcomes. If the Pathways to Healthcare program has an 
impact on earnings, then it might also be expected to affect these outcomes. Even without an 
earnings impact, there may be impacts on some of these well-being measures. For example, 
the financial assistance component of the program may lower school debt levels. 


Chapter 7 concludes with a discussion of the findings and open questions for future 
research. 


A separate Appendix volume provides technical details on analysis methods, data sources, and 
sensitivity analysis. A forthcoming cost addendum will describe the costs of the Pathways to 
Healthcare program. 
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This chapter describes the PACE evaluation design and analytic methods as applied to the 
Pathways to Healthcare program. It begins with a discussion of the program’s theory of change. 
It then describes the evaluation design, data sources, and analysis procedures. 


2.1 Pathways to Healthcare Theory of Change 


Exhibit 2-1 below depicts the career pathways theory of change as applied to Pathways to 
Healthcare. It shows in detail how the program is hypothesized to produce effects on outcomes 
such as career knowledge and resources, which in turn will lead to effects on early (18-month) 
outcomes such as hours of training and credential receipt, and eventually to longer-term gains 
in main outcomes, such as employment, earnings, additional credentials, and other life 
outcomes. 


Starting in the box at the left, the theory of change begins with program inputs and program 
components. The earlier, short-term report (Gardiner et al. 2017) found that these program 
inputs (PCC and PCOS services, PCC program staff and leadership, HPOG funding, and 
participant characteristics) and program components (assessment, instruction, supports, and 
employment connections) were largely in place and operating as anticipated. *° 


The middle box shows the intermediate outcomes. Improving participants’ competencies and 
career knowledge, removing barriers to school or work, and addressing life challenges was a 
necessary precursor to improving the main (longer-term) outcomes of interest. The Pathways to 
Healthcare program intended to affect these outcomes quickly so that students would be better 
positioned to engage in education and training. As described in Chapter 1, the short-term report 
found improvements in some but not all of these areas. 


= Although the Pathways to Healthcare program ended in 2016, PCC continues to provide occupational 


training through its Center for Training and Development, and PCOS (now called ARIZONA@WORK 
Pima County) still provides employment services through the Workforce Innovation and Opportunity 
Act. In addition, PCC received a second-round HPOG grant to operate the HOPES program, which 
retains many Pathways to Healthcare components. 


The HOPES program is being evaluated as part of ACF’s National Evaluation of the 2nd Generation 
of Health Profession Opportunity Grants (HPOG 2.0 National Evaluation). Although random 
assignment governs access to the HOPES program, OPRE and the evaluation team did not want any 
PACE study participants to be randomly assigned a second time. Therefore, Pathways to Healthcare 
participants can bypass random assignment and participate in HOPES services. PACE control group 
members could not receive Pathways to Healthcare program services for three years following 
random assignment. After this embargo period, control group members could enroll in HOPES 
without going through random assignment. More information about the HPOG 2.0 National Evaluation 
is available at https://www.acf.hhs.gov/opre/research/project/national-evaluation-of-the-2nd- 
generation-of-health-profession-opportunity-grants-hpog-20-national-evaluation. 
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Exhibit 2-1: Career Pathways Theory of Change for Pathways to Healthcare 


PROGRAM INPUTS INTERMEDIATE OUTCOMES MAIN OUTCOMES 


ORGANIZATION 
* PCC and PCOS 
* CTD infrastructure 


* HPOG-funded 
instructors 


* PCOS and PCC 
advisors 


* Program leadership 
* HPOG funding 


PARTICIPANTS 
+ Low-income 


+ Resident of 
Pima County 


* Interest in 
healthcare career 


PROGRAM COMPONENTS 


ASSESSMENT 
* TABE 
* Compass 


+ Individual 
Employment Plan 


INSTRUCTION 


* 16 healthcare 
programs 
organized into 5 
pathways 


+ Accelerated and 
contextualized 
basic skills course 


* Accelerated basic 
skills open lab 


+ Work-based 
learning 


Abt Associates 


SUPPORTS 


* Career exploration/ 
training plan 


+ Academic advising 
+ Personal advising 


+ Assistance 
removing barriers 


+ Financial assistance 
(scholarships) 


* Instructional 
supports 


EMPLOYMENT 


+ Employability skills 


* Post-training job 
search assistance 


+ Networking group 


LOCAL POSTSECONDARY 
TRAINING SYSTEMS 


* Competing training 


programs 


GENERAL (21ST CENTURY) COMPETENCIES 
+ Improved basic academic skills 


+ Improved psycho-social skills (persistence, 
academic self-confidence, self-evaluation, 
sense of belonging) 


SPECIFIC COMPETENCIES 


* Improved occupational skills in desired 
healthcare area 


CAREER KNOWLEDGE 


+ Increased awareness of steps needed to 
reach career goals 


+ Understanding of required skills and 
responsibilities associated with various 
healthcare occupations 


* Increased knowledge of labor market 


RESOURCES 


* Constraints addressed through financial 
assistance and supportive services 


LIFE CHALLENGES 


+ Reduced financial hardship 
+ Reduced stressors 


CONTEXTUAL FACTORS 


|Rolor.\ ii ere) Te] ih 4 
* Healthcare job openings 
* Growth in healthcare jobs 


POSTSECONDARY ATTAINMENT 

* Hours of training received 

* Credits 

* Credentials 

* Engage in next step on the pathway 


SUCCESSFUL IN CAREER-TRACK 
EMPLOYMENT 


* Obtain employment in healthcare occupation 
+ Increased earnings 
* Perceived career progress 


OTHER LIFE OUTCOMES 


* Improved dual we ng 
* Improved family economic status 


OTHER COMMUNITY FACTORS 
* Size of target population 

+ Other service providers 

* Referral partners 
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The main outcomes, which are the focus of this report, appear in the far right box. They include 
additional postsecondary credential attainment, career-track employment, and improvement in 
other life outcomes, such as family economic well-being. These outcomes are most directly 
connected to the program’s goal of improving employment and earnings for TANF recipients 
and other low-income individuals. The earlier, short-term report highlighted the impact of 
Pathways to Healthcare on postsecondary attainment after 18 months. However, a non-trivial 
proportion of students in both the treatment and control groups were still enrolled in training. As 
a result, we anticipated that educational impacts might continue to evolve. For example, 
perhaps members in the treatment group will get differentially more training; perhaps those in 
the control group will catch up. This report documents impacts on postsecondary attainment 
again after three years. 


Aside from some measures of career progress and job quality, the short-term report did not 
include impacts on earnings and employment, anticipating that it was too early to draw 
conclusions at that time. However, with many treatment group members participating in 
relatively short-duration training programs, it Seems reasonable to expect impacts to emerge 
within three years. The theory of change also specifies that if improvements in educational 
attainment lead to improvements in earnings and employment, then that should in turn lead to 
improvements in other life outcomes. It seems reasonable that these changes should be visible 
now, three years out. 


Finally, the exhibit shows that a number of contextual factors can affect impacts including 
other available training programs and local economic conditions. The earlier, implementation 
study explored these factors (See Gardiner et al. 2017), and we discuss them again in this 
report when they are useful for explaining program impacts. 


2.2 Research Questions at Three-Year Follow-up 

Three years after random assignment, what were the effects of Pathways to Healthcare on: 
e educational outcomes? 
e entry into career-track employment and higher earnings? 


e individual and family well-being, including income and other life outcomes? 
Each of these research questions is addressed, in turn, in the chapters that follow. 


2.3 Data Sources 


Analyses in this report use several data sources: PCC records; baseline surveys administered 
to study participants immediately prior to random assignment; follow-up surveys conducted 
approximately 18 months and three years after random assignment; earnings and employment 
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data from the National Directory of New Hires (NDNH); and school enrollment data from the 
National Student Clearinghouse (NSC).** We describe each of these data sources below. 


2.3.1 Local College Records 


PCC records are the chief data source for education outcomes in this report. These records 
measure participation in education and training at PCC for both the treatment and control 
groups, as well as receipt of credentials. We looked at the degree to which PCC enrollments 
covered college experience in the sample by checking against NSC records. We adjusted for 
modest differences found, using NSC data (see Section 2.3.4 for description of NSC data).*° 


2.3.2 Baseline Surveys 


This report uses data from the baseline surveys to describe the sample and for regression 
adjustment. All 1,217 study participants completed the Basic Information Form just prior to 
random assignment. This form captured demographic information, family characteristics, 
educational history, and work and earnings information. At this time study participants also 
completed a Self-Administered Questionnaire, which collected more-sensitive personal 
information such as training commitment and academic confidence. *° 


2.3.3 Follow-up Surveys 


This report focuses on outcomes measured in a three-year follow-up survey, with some 
reference to 18-month follow-up survey data analyzed in the short-term report. 


18-month Survey. The earlier follow-up survey provided measures of outcomes that the theory 
of change indicated Pathways to Healthcare might affect in the short term. Administered by 
telephone or in person, the 18-month survey response rate was 80 percent (82 percent in the 
treatment group and 78 percent in the control group). Some of the findings summarized in 
Chapter 1 are based on these data. The other use of the 18-month survey data in this report is 
to help impute values for missing data on job and education spells from other data sources. 


Three-year Survey. We designed the three-year follow-up survey to measure outcomes that 
the theory of change indicated Pathways to Healthcare might affect over a longer time horizon, 
such as employment and other life outcomes. The survey also captured detail on respondents’ 
educational history, a limited number of psycho-social skills, and their children’s experiences 


4 To learn about HPOG1.0 and PACE restricted use data archived at the Inter-university Consortium 


for Political and Social Research, see: https://doi.org/10.3886/ICPSR37290 (for HPOG) and 
https://doi.org/10.3886/ICPSR37289 (for PACE). 
See Appendix B for more details on our decision to use PCC records for education outcomes and the 


construction of educational outcome measures. See Appendix E for sensitivity tests for outcome 
measures from the different potential data sources. 


35 


3° PCC staff administered the Basic Information Form on paper and then entered it electronically into 


the study database. Because the Self-Administered Questionnaire asked for personal information 
(criminal records, psycho-social skills, social support, career orientation and knowledge, and personal 
and family challenges), study participants filled out a paper form and then placed it in a sealed 
envelope that PCC staff sent to Abt Associates for data entry. 
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with school (as applicable). The response rate for the three-year follow-up survey was 86 
percent overall (87 percent in the treatment group and 84 percent in the control group). The 
median response occurred at 38 months.*’ Appendix C provides detailed descriptions of the 
outcomes based on the three-year survey used in this report.*® 


2.3.4 National Student Clearinghouse 


This study used data on college enrollment from the NSC to evaluate and adjust local college 
records and to analyze and adjust for the survey (see Appendix D). NSC is a nonprofit 
organization that collects data on student enrollment, degrees earned, and other credential 
completion data from most U.S. institutions of higher education. Designed to aid the 
administration of student loan programs, researchers also use NSC data to study college 
access and persistence. As in most administrative data systems, data are subject to various 
coverage and content limitations. Most importantly, coverage of private, for-profit two-year 
colleges is very low (less than 30 percent), and the NSC makes no attempt to collect data from 
schools that are not colleges (i.e., not accredited to grant degrees). 


2.3.5 National Directory of New Hires 


Wage records from the NDNH are a major data source for earnings and employment analyses 
in this report. Maintained by the federal Office of Child Support Enforcement, the NDNH 
includes quarterly earnings measured by state Unemployment Insurance systems and earnings 
of federal civilian and military employees provided by various federal agencies. The PACE 
evaluation had access to these data for study sample members for two years prior to random 
assignment through the end of the evaluation period.*° Additional detail is provided in 

Appendix F. 


2.4 Evaluation Design and Analysis Plan 


The PACE evaluation used an experimental design to estimate the impact of access to 
Pathways to Healthcare on participants’ outcomes. Such a design ensures that any estimated 
impacts can be attributed to program access rather than to unmeasured differences between 
eligible study sample members with access (the treatment group) and without access (the 
control group). 


37 The median response occurred at 38 months. More than 75 percent of the respondents completed 


the survey 39 months or less after random assignment. The longest lag between randomization and 
completion was 53 months. Additional months of follow-up potentially increases recall error and shifts 
means for time-sensitive variables. However, the lags were fairly well balanced between the 
treatment and control groups, so this variation in lags between randomization and completion should 
not lead to false claims of program effects. 


The full instrument is available at http://www.career-pathways.org/career-pathways-pace-three-year- 
instrument/. 


38 


3° At the time this three-year impact report was written, 18 quarters of NDNH data were available. 


However, the pre-specified confirmatory and secondary outcomes in this report use only the first 13 
quarters. 
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As designed, the experiment captured impacts for all sample members, regardless of whether 
those assigned to the treatment group actually received the program services. In other words, 
this design—an “intent to treat” approach—assesses whether access to the program including 
all of its components led to better outcomes for those offered the chance to participate in it, 
relative to what they could have obtained without the program. For a voluntary (rather than 
mandatory) program, the intent to treat estimate is often the most policy relevant. However, it is 
important to remember that the program is being compared to other programs and services that 
are available in the local area, rather than being compared to no training. 


Another important aspect of the PACE evaluation’s design is that the experiment captures the 
effects of the program overall, rather than the contributions of its components. Designers of 
Pathways to Healthcare deliberately included a package of multiple strategies (e.g., 
assessment, instruction, supports, and employment connections) that they hypothesized were 
needed to produce desired impacts. As a result, the evaluation focuses on whether the program 
as a whole, when implemented in real-world conditions, produces an impact. 


2.4.1 Hypothesis Testing 


The Pathways to Healthcare theory of change targets a range of outcomes. The PACE 
evaluation structures the analysis by establishing three categories of hypotheses: 


e Confirmatory hypotheses center on outcomes most critical to judging the program's 
success in achieving its goals within the designated time period. By limiting the 
confirmatory analysis to a single outcome in each of two separate domains, we avoid the 
statistical problem that arises from “multiple comparisons.”“° For the three-year impact 
study of Pathways to Healthcare, we specified two confirmatory tests: an increase in 
attainment of postsecondary credentials requiring a year or more of college in the 
education domain and an increase in average quarterly earnings in quarters 12 and 13 
after random assignment in the employment domain. Because each has a hypothesized 
direction (an increase in the average level of the outcome) we applied a one-tailed test 
of statistical significance, ignoring possible effects in the other direction. 


e Secondary hypotheses address a parsimonious set of other important indicators of 
program success. Secondary hypotheses also posit effects in an expected direction, so 
we applied one-tailed tests for statistically significant effects only in the specified 
direction. Outcomes for these hypotheses at three-year follow-up include credential 
receipt, enrollment and number of college credits, employment status and indicators of 
career pathways employment, indicators of career progress, and measures of financial 
well-being. The hypothesized direction is an increase in the average level for all 


a Testing for program impacts on so many outcomes causes a statistical problem: it provides the 


program many chances to demonstrate success, and with enough chances even an unsuccessful 
program might appear to have one or two impacts. If the evaluation did not account in some way for 
the multiplicity of hypothesis tests, some of findings would reach conventional levels of statistical 
significance merely by chance, even if there were no real effects on any outcome. This is known as 
the problem of “multiple comparisons.” 
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outcomes other than some measures of financial distress, where we hypothesize a 
decrease in the average level. 


e Exploratory hypotheses include a larger number of additional possible effects for 
related outcomes. They are intended to help improve our understanding of findings from 
the confirmatory and secondary analyses. Exploratory hypotheses may but do not 
necessarily speculate the direction of effects, and therefore we applied two-tailed tests. 
Some examples of outcomes for exploratory hypotheses include quarterly earnings and 
employment for each quarter after random assignment, various measures of job quality, 
and measures of financial well-being such as household income. 


Prior to estimating Pathways to Healthcare impacts, the research team published an analysis 
plan specifying key hypotheses and outcome measures.** The team subsequently assessed 
data quality, refined the plan, and publicly registered it on the Open Science Framework 
website.** The purpose of the analysis plan and registration was to guide the work of the 
research team and publicly commit to particular hypotheses and an estimation approach that 
aligns with ACF’s commitment to promote rigor, relevance, transparency, independence, and 
ethics in the conduct of evaluations.*° 


2.4.2 Impact Estimation Procedures 


We conducted analyses to estimate the impact of Pathways to Healthcare on the hypothesized 
outcomes described above. 


Random assignment ensures that, on average, study sample members in the treatment and 
control groups will have similar characteristics at baseline. Random assignment also ensures 
that measured differences in subsequent outcomes provide unbiased estimates of program 
impacts. To address any effects that chance differences arising from random assignment might 
have on estimates, analysts typically estimate impacts using a procedure that compensates for 
chance differences in measured baseline characteristics. Such procedures also help to increase 
the precision of estimates. 


To select baseline characteristics and estimate impacts, the PACE evaluation developed an 
approach that respects the conservative tradition of including out-of-balance characteristics, no 
matter what, in addition to empirically selected covariates, but without incurring large losses in 
precision. We describe details of this approach, a recently developed technique called “least 
absolute shrinkage and selection operator” (LASSO), in Appendix A. 


We then used a regression-adjustment model—including the identified covariates—to estimate 
impacts. All analyses of survey data applied weights developed to adjust for differential 
nonresponse across groups of study participants that have different likelihoods of survey 


“* See Judkins, Fein, and Buron (2018). 
42 


See https://osf.io/wj6gc/ for the 18-month report registration and https://osf.io/ua4bw/ for the three- 
year report registration. 


See https://www.acf.hhs.gov/opre/resource/acf-evaluation-policy. 
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response. Additional details on these and other aspects of the analysis appear in Appendices A 
and B. 


The text box How to Read Impact Tables describes how to navigate and understand the tables 
in the impact chapters. 


How to Read Impact Tables 


The exhibits in Chapters 4-6 show the outcome measure in the left-most column (Outcome). 


The next column (Treatment Group) presents the treatment group’s regression-adjusted mean 
outcome, followed in the next column by the control group’s actual mean outcome (Control 
Group). The regression adjustments correct for random variation in baseline covariates between 
the two groups (and thus differ slightly from the raw means) and improve the precision of the 
estimates. 


The next column (Impact (Difference)) is the impact of being offered Pathways to Healthcare— 
that is, the difference between the treatment and control group means. The Standard Error 
column is a measure of uncertainty in the estimated impact that reflects both chance variation due 
to randomization and any measurement error. The column labeled Relative Impact presents the 
impact as a percentage change from the control group mean. It offers a sense of how “big” or 
“small” the impact on the treatment group is, at least relative to the control group’s level. For 
outcomes with no natural unit of measurement we report an Effect Size instead of the relative 
impact. The effect size is a standardized measure that defines impacts as a fraction of the pooled 
standard deviation across the treatment and control groups. It offers a sense of the size of the 
impact relative to how much the outcome varies across the full sample and allows for comparison 
of the size of the impact across scale outcomes. 


The final column, p-Value, is the probability that the observed or a larger difference between the 
treatment and control groups would occur by chance, even if there was in reality no difference 
between the two groups. 


Statistical significance 


There are several common standards for judging statistical significance. In this report, tests are 
considered statistically significant and highlighted in tables if the p-value is less than .10. The 
smaller the p-value, the more likely that the observed difference between the treatment and control 
groups is real, rather than occurring by chance. Tests with p-values smaller than .10 are separately 
flagged: 


*for.10 (10 percent level) 


**for .05 (5 percent level) 


** for .01 (1 percent level) 


Categories of findings 


Tests of statistical significance for confirmatory and secondary outcomes are one-sided tests 
because we have a directional hypothesis for these impacts. The confirmatory and secondary 
analyses are reported using bold text in the tables. Tests of significance for exploratory outcomes 
use a two-sided test, a test we use because we do not have a directional hypothesis. Exploratory 
analyses are reported using regular (not bolded) text in the tables. 
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Pathways to Healthcare was designed to facilitate multiple occupational training steps in each of 
five pathways. Students who needed to improve their basic skills first enrolled in College 
Readiness. Thereafter, each pathway spanned entry-level programs (Level 1) through 
advanced-level ones (Level 3), as described in Chapter 1 (see Exhibit 1.1). This chapter 
describes the treatment group’s engagement in training beyond the early follow-up period. 


As of the short-term follow-up period, 48 percent of treatment group members had enrolled in 
occupational training: 35 percent enrolled directly in occupational training, and 13 percent 
enrolled in occupational training after completing a College Readiness bridge. Among the 52 
percent who did not enroll in occupational training, 38 percent did not engage in any services, 
and the remaining 13 percent enrolled in and completed a College Readiness bridge but did not 
progress to training. 


Section 3.1 explores the extent to which treatment group members made progress on their 
pathways over the three-year follow-up by advancing at least one level. Advancement through 
the pathways is key to the theory of change, and detailed information on courses of study and 
credentials are only available for the treatment group.“* For instance, for those who start in a 
Level 1 training to prepare for an entry-level job, progression to subsequent levels on the 
pathway is likely needed to raise earnings. Occupations associated with Level 1 credentials 
have entry-level wages comparable to other entry-level occupations outside of healthcare. The 
program could only expect to increase wages if it moved more treatment group members to 
higher levels on the pathways than they would otherwise have accessed. 


Section 3.2 focuses on the longer-term follow-up period (between the 18-month and three-year 
follow ups) and identifies the credentials earned in terms of level and length. The specific 
credentials give more detail on the particular pathways participants pursued. Section 3.3 
examines whether treatment group members who did not engage in training in the first follow-up 
period did so by the time of the second follow-up at three years. 


Section 3.4 explores treatment group participation in PCC’s second-round HPOG-funded 
program, HOPES. After Pathways to Healthcare ended in 2016, HOPES became the available 
source of HPOG tuition assistance and other supports. 


Section 3.5 summarizes the participation findings, which in turn set the stage for the impacts on 
postsecondary training in Chapter 4. 


3.1 Movement along the Pathways 


Our first question is whether treatment group members who enrolled in any training by 18 
months continued to another pathway level by three years after they were randomly assigned. 


“4 Administrative records indicate whether the control group was enrolled in PCC or not, but we do not 


have more detailed information (e.g., the particular course of study) for control group members. 


‘Abt Associates 
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Overall, 37 percent of the treatment group earned a credential within three years. Almost one 
third of the credentials earned were at least one level higher than the initial training level earned 
by 18 months, indicating that these participants took a second step on their pathway. 


Exhibit 3-1 below reflects this movement using a flow diagram. The starting points on the left of 
the exhibit reflect the level of the first PCC training in which treatment group members enrolled, 
and the endpoints on the right of the exhibit reflect the level of the highest credential they 
earned through three years. The gray flows reflect those whose first PCC training was at the 
College Readiness level; the blue flows reflect those whose first PCC training was at Level 1; 
the red flows reflect those whose first PCC training was at Level 2; and the green flows reflect 
those whose first PCC training was at Level 3. The widths of each start point, endpoint, and the 
flows from left to right are all proportional to the number of participants who followed that flow. 


As reflected in Exhibit 3-1, starting at the top left side and moving down the exhibit: 


e 26 percent of all treatment group members started their training at the College 
Readiness level. Nine (9) percent of all treatment group members earned a credential by 
three years after random assignment having started at College Readiness: 3 percent 
earned a Level 1 credential; 5 percent earned a Level 2 credential; and 1 percent earned 
a Level 3 credential. 


e Moving down the exhibit, 20 percent of all treatment group members started their 
training at Level 1. The large majority of them (17 percent of all treatment group 
members) earned credentials within three years: 8 percent earned a Level 1 credential 
only; 7 percent earned a Level 2 credential; and 2 percent earned a Level 3 credential. 


e Fewer treatment group members (9 percent of all treatment group members) started 
their training at Level 2. Six (6) percent of all treatment group members earned a Level 2 
credential within three years, and fewer than 1 percent earned a Level 3 credential. 


e Seven (7) percent of all treatment group members started their training at Level 3. About 
half of them earned credentials within three years: fewer than 1 percent of all treatment 
group members earned a Level 2 credential, and 3 percent earned a Level 3 credential. 


Taken together, by three years after random assignment, almost 19 percent of all treatment 
group members had earned a credential at least one level higher than the level where they 
began their training: 


e the 9 percent who started in College Readiness transitioned to Level 1, 2, or 3; 
e the 9 percent who started in a Level 1 course went on to Level 2 or 3; and 


e the <1 percent who started in a Level 2 course went on to Level 3. 


This 19 percent of all treatment group members represents 31 percent of those who enrolled in 
any education or training within the first 18 months of follow-up. 
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Exhibit 3-1: Flows from First Training to Credentials over a Three-Year Follow-up Period 
(Treatment Group Only) 


Level of First PCC Training Experience by Month 18 Highest Credential Earned by Three Years 


No Occupational Credential 
(25%) 


College Readiness 
(26%) 


Level 1 Credential 
(11%) 


Level 1 
(20%) 


Level 2 Credential 
(19%) 


Level 2 
(9%) 


Level 3 
(7%) 


Level 3 Credential 
(7%) 


Source: PCC records. 

Note: Starting points on the left side of the figure reflect the level of the first PCC training in which participants enrolled in the first 18 months 
after random assignment. (It does not include the 6 percent all of treatment group members who started their initial training in month 19 or 
later.) Endpoints on the right side of the figure reflect the level of the highest credential earned by the end of the three-year follow-up period. 
Starting point and endpoint percentage labels reflect percentage of the entire treatment group of 609. Almost 62 percent of treatment group 
members (375 of 609) enrolled in any training within 18 months. 
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3.2 Receipt of Long-Duration Credentials between the Two Follow-ups 


Some treatment group members earned credentials taking a year or more of study (“long- 
duration” credentials) during the first 18 months of follow-up, but more did so between the 18- 
month and three-year follow-ups. 


PCC records show that 18 months after random assignment, 3 percent of treatment group 
members had earned a long-duration credential. The most common long-duration credentials 
earned were for Licensed Practical Nurse (LPN) (Level 3) and Medical Assistant (Level 2), each 
accounting for one third of those credentials attained. The remaining one third included Level 2 
Surgery Technician, Level 3 Pharmacy Technician, and associate degrees not included ona 
pathway (Liberal Arts). 


A larger share of treatment group members, 11 percent, earned a long-duration credential in 
months 19 to 36, and most of these were Level 2 or 3 credentials. PCC records show that 
Medical Assistant (Level 2) was the most common, accounting for 44 percent of those 
credentials, followed by LPN (31 percent). About 4 percent of long-duration credentials were for 
Pharmacy Technician (Level 3) or Surgical Technician (Level 2). Sixteen percent of long- 
duration credentials were Associate of Applied Science degrees or Associate of Arts degrees. 


3.3 ‘Initial Engagement with Occupational Training after Short-Term Follow-up 


We also explored whether the 52 percent of treatment group members who did not enroll in 
occupational training during the first follow-up period did so by three years after random 
assignment. The short-term follow-up survey responses indicated lack of time and money 
contributed to non-enrollment decisions. As well, the delay between study enrollment and the 
start of training—an average of four months according to PCC administrative records—could be 
a factor. 


We found 8 percent of the treatment group enrolled in occupational training for the first time 
after the short-term follow-up. This includes 2 percent of those who completed College 
Readiness during the initial follow-up period and transitioned to occupational training after 18 
months and the 6 percent who began directly in occupational training after month 18. 
Collectively, this group earned 3 percent of the credentials earned by the treatment group as a 
whole during the second follow-up period. 


Although a modest number of additional treatment group members first engaged in occupational 
training between months 18 and 36, more than 40 percent of all treatment group members had 
still not engaged by the three-year follow-up. 
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3.4 Continued Training through HPOG 2.0 Grant 


To reduce the financial burden of attending school, Pathways to Healthcare provided 
scholarships for occupational training. Pathways to Healthcare’s scholarships, advising, and 
other services intended to help students persist ended in 2016 when PCC’s HPOG 1.0 grant 
ended. 


Although Pathways to Healthcare services were no longer available to support their training 
when the first round HPOG grant ended, treatment group members could receive services 
through PCC’s second-round HPOG grant, HOPES. Although not identical, HOPES retains 
many Pathways to Healthcare components, including distinct career pathways, scholarships, 
advising, and employment supports. Like Pathways to Healthcare, HOPES is part of a 
randomized controlled trial evaluation; however, as noted earlier, ACF allowed Pathways to 
Healthcare study participants to bypass that evaluation’s random assignment and receive 
HOPES services.” Control group members, too, could bypass random assignment once their 
three-year embargo from receiving Pathways to Healthcare program services ended.*° 


HOPES data show that a small share of Pathways to Healthcare study participants did pursue 
additional training.*’ According to HOPES program records, 84 treatment group members, or 
nearly 14 percent of the treatment group, received training as part of HOPES. As well, 34 
control group members, or 6 percent of the control group, bypassed random assignment and 
enrolled in HOPES services. 


3.5 Summary 


This chapter provided a detailed, descriptive overview of occupational training receipt within the 
treatment group. Analyses in this chapter show that within three years of random assignment, 
about one in five treatment group members (19 percent) earned a credential at least one level 
higher than where they began their training. About 11 percent of the treatment group earned 
these credentials after the short-term follow-up at 18 months, and many of the credentials they 
earned were for programs that took a year or more of study. This is the type of progress that 
should lead to occupations with higher starting wages, compared with treatment group members 
who earned Level 1 credentials only or no credentials at all. More than 40 percent of the 
treatment group had not engaged in any training by 36 months. 


The next chapter reports estimates of the Pathways to Healthcare program’s impact on 
postsecondary training; that is, estimates for the treatment group relative to the control group. 


“S OPRE and the evaluation team did not want any PACE study participants to be randomly assigned a 


second time. 


“© As of January 2017, all control group members could seek HOPES services. 


“7 These tabulations come from the management information system developed for the HPOG 2.0 


Program in use by all second-round HPOG grantees, called the Participant Accomplishment and 
Grant Evaluation System (PAGES). Data in PAGES are available for members of the Pathways to 
Healthcare treatment and control groups who enrolled in HOPES after Pathways to Healthcare 
ended. 
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As described in that chapter, more than 9 percent of treatment group members were still 
enrolled in college at the end of the three-year follow-up period. Some of these treatment group 
members accessed training and supports through HOPES, the HPOG 2.0 follow-on program to 
Pathways to Healthcare that is not expected to end until 2020. Regardless of the source, 
movement through a program’s pathways to higher-level credentials is a key feature of the 
career pathways theory of change and necessary to observe impacts on earnings. 
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This chapter reports the impact of Pathways to Healthcare on postsecondary training for the 
three-year follow-up period, with some analyses extending to nearly five years after random 
assignment. Whereas Chapter 3 focused on the experiences of the treatment group only, this 
chapter assesses the Pathways to Healthcare program’s impact by comparing the treatment 
group’s versus the control group’s training outcomes. 


The Pathways to Healthcare theory of change posits that the combination of basic skills 
remediation (College Readiness bridges, for those who need it) and occupational training, 
coupled with a range of academic and non-academic supports, will increase postsecondary 
credential attainment. 


Though credential attainment and career-track employment are Pathways to Healthcare’s 
ultimate goals, it seemed likely based on PCC’s program model that a large proportion of 
students would still be engaged in training at 18 months. As a result, the pre-specified 
confirmatory outcome for the 18-month follow-up period was hours of occupationally focused 
training.*® Indeed, about 40 percent of the treatment group and 25 percent of the control group 
were still enrolled in college occupational training in the last six months of the 18-month follow- 
up period. The short-term report concluded that the treatment group received more total hours 
of occupationally focused training and more total credentials than the control group, indicating 
the program was on the right track toward achieving those ultimate goals (Gardiner et al. 2017). 


By three years after random assignment, it is reasonable to expect completion of “long-duration” 
credentials (those taking a year or more of college to earn, which include associate or higher 
degrees). Thus, the confirmatory outcome for the education domain in this report is receipt of 
any postsecondary credential that typically requires at least a year of college to earn. The theory 
of change suggests that this outcome is appropriate for assessing whether Pathways to 
Healthcare is continuing to meet its postsecondary attainment goals after three years. The 
increase in hours of training and credentials in the short term, coupled with high rates of 
continued enrollment, positions the program for impacts on long-duration credentials associated 
with Level 2 or Level 3 training within each pathway. 


This chapter uses PCC records and study participants’ responses to the three-year follow-up 
survey to report impacts on credentials as well as on enrollment and credits.“° 


“8 Hours of training completed appeared to be the best single indicator of whether the program was 


meeting its short-term goal, per its theory of change. The measure included both credit training and 
noncredit training, as the five healthcare pathways included both types. 


“See Appendix B for a discussion of imputation procedures to incorporate data from the NSC and the 


three-year follow-up survey for those who enrolled in schools other than PCC. 
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4.1 Impact on Credentials 


This section describes impacts on credential receipt. It addresses two related, but distinct, 
issues. First, the chapter reports the impact on total credentials received after random 
assignment. It begins with the three-year confirmatory outcome, receipt of a postsecondary 
credential taking one or more years of college to earn. This longer-term postsecondary 
milestone was attainable within the follow-up period, even for treatment group members who 
started at the College Readiness level or control group members who started with basic skills 
remediation. We also assess impacts for receipt of other types of credentials. Second, the 
chapter considers how impacts for credentials have changed since the 18-month follow-up. 


m The treatment group earned more credentials that take at least a year of college 
than the control group did. 


Pathways to Healthcare increased the receipt of credentials typically requiring a year or more of 
college from 11 percent to 18 percent, an impact of 7 percentage points (Exhibit 4-1). This 
impact represents a 62 percent increase over the control group. 


= Pathways to Healthcare had larger impacts on receipt of postsecondary 
credentials defined broadly than it did on long-duration credentials specifically. 


The theory of change predicts not only an increase in long-duration credentials three years after 
random assignment, but also that impacts on short-duration credentials should grow. This is 
particularly true given the rates of continued enrollment at the conclusion of the 18-month 
follow-up period. The theory of change implies that impacts would grow as participants engaged 
in and completed initial or follow-on credentials.°° Indeed, Pathways to Healthcare increased 
receipt of short-duration credentials by 20 percentage points. The impact for any credential 
(which includes both short-duration and long-duration ones) is about three times as large as the 
impact for long-duration credentials only (see Exhibit 4-1).°* Most credentials earned were 
healthcare related for both the treatment and control groups.°” 


The impact on credentials was not limited to those received from colleges. Consistent with the 
short-term impact report, treatment group members received more credentials from any type of 
school (a college, another education/training institution, or a licensing/certification body) than did 
the control group. A greater share of the treatment group earned any credential (21 percentage 


°° The most common Level 2 program during the 18-month follow-up period was Patient Care 


Technician—a short-duration, add-on credential to Nursing Assistant. 


+ This finding is also corroborated using responses to the three-year follow-up survey. In those 


analyses we find an impact of 18 percentage points for receipt of any type of postsecondary 
credential from a college, and an impact of 5 percentage points for receipt of any postsecondary 
credential that typically requires at least a year of college. Both impacts are statistically significant at 
the 5 percent level. 


°? Exhibit 4-1 notes reports that 40 percent of the treatment group earned any postsecondary credential 


from a college. This rate differs from the figure reported in Chapter 3 (37 percent). This difference is 
due to the imputation procedures we use in this chapter and that these estimates are regression- 
adjusted impacts, whereas Chapter 3 reported rates from PCC records that are not regression 
adjusted. 


Associates 
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point impact) or a healthcare credential (22 percentage point impact) from any type of school 
than did the control group. This suggests that control group members did not substitute another 
type of training provider for a college. The only outcome for which there was no detectable 
impact was receipt of an exam-based certification or license. 


Exhibit 4-1: Three-Year Impacts on Postsecondary Credentials 


Treatment Control Impact Standard Relative 
Outcome Group Group __ (Difference) Error Impact _p-Value 

PCC Records 

Confirmatory Outcome: Received credential 

taking 1+ year of college (%) 18.4 11.3 +7,0*** (2.0) +61.9 <.001 

Received at least associate degree (%) 4.3 2.5 +1,.8* (1.0) +72.0 .073 

Received any credential from a college (%) 39.9 16.4 +23.5*** (2.4) 4143.3 <.001 

Received healthcare credential from a college 

(%) 38.0 14.1 +23.9*** (2.4) +169.5 <,001 

Received credential taking <1 year of college (%) 26.0 5.6 +20.4*** (2.0) +3643 <.001 
Sample size 609 608 

Received exam-based certification or license (%) 30.3 27.8 +2.4 (2.9) +8.6 .203 

Three-year Follow-up Survey and PCC Records 

Received a credential from any type of school (%) 44.2 22.8 +21.4*** (2.9) +93.9  <,001 

Received healthcare credential from any type of 

school (%) 41.5 19.4 +22.1*** (2.8) +113.9 <.001 
Sample size 530 510 

Number of credentials received from any type of 

school among those who earned credentials (non- 

experimental) (#) LAL 1.48 +0.22** (0.09) +14.9 .019 
Sample size 244 113 

Number of healthcare credentials received from any 

type of school among those who earned credentials 

(non-experimental) (#) 1.67 1.37 +0.30*** (0.09) +21.9  <.001 
Sample size 229 95 


Source: PCC records: postsecondary credentials. Blend of PCC and three-year follow-up survey data: “any type of school” credentials. Blend 
of 18-month and three-year follow-up survey data: exam-based credentials. 


Note: Confirmatory and secondary outcomes are bolded and statistical significance is based on one-tailed tests; exploratory outcomes are 
not bolded and statistical significance is based on two-tailed tests. Italicized outcomes identify comparisons that are non-experimental. 
“Relative Impact” represents impacts as a percentage of the corresponding control group mean (i.e., 100 x [impact/control group mean). 


Statistical significance levels based on differences between research groups: *** 1 percent level: ** 5 percent level; * 10 percent level. 


= The impact of Pathways to Healthcare on long-duration credentials was larger in 
months 19 to 36 after random assignment than in the first 18 months. 


Exhibit 4-2 below compares impacts on credentials over time. The levels for credentials that 
take a year or more of college to earn are quite small in both the treatment and control groups in 
the first 18 months, which supports the decision to focus on training hours in the short-term 
report. In fact, there is a negative impact on receipt of such long-duration credentials. 


The impact both for any credential taking a year or more of college and specifically for associate 
degrees or higher was larger in months 19 to 36 than in the first 18 months. Statistical tests 
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conclude that the impacts for receipt of the two outcomes over months 19 to 36 differ from the 
impacts for those same two outcomes over the first 18 months. Over the later months, 
Pathways to Healthcare increased receipt of any credential taking at least a year of college by 9 
percentage points and receipt of associate degrees or higher by 2 percentage points. 


Exhibit 4-2: Impacts on Postsecondary Credentials over Time 


Treatment Control Impact Standard _ Relative 
Outcome Group Group (Difference) Error Impact p-Value 

Months 1-18 

Received credential taking 1+ year of 

college (%) 3.4 5.4 -2.0* (1.2) -37.0 .084 
Received at least associate degree (%) 1.0 1.2 -0.1 (0.6) -8.3 821 
Received any credential from a college (%) 23.5 10.2 +13,3** (2.1) +130.4 <.001 
Received healthcare credential from a 

college (%) 23.0 8.7 +14.3** (2.1) +1644 <.001 


Months 19-36 


Received credential taking 1+ year of 


college (%) 15.3 6.2 +9.0*"* (1.7) +145.2 = <.001 

Received at least associate degree (%) 3.3 13 +1.9 * (0.8) +146.2 .020 

Received any credential from a college (%) 20.7 6.7 +14.0** (1.9) +208.9 <.001 

Received healthcare credential from a 

college (%) 19.1 5.8 +13.3** (1.8) +2293 = <.001 
Sample size 609 608 


Source: PCC records. 


Note: Statistical significance is based on two-tailed tests. “Relative Impact” represents impacts as a percentage of the corresponding control 
group mean (i.e., 100 x [impact/control group mean]). Some people earned credentials in both months 1-18 and months 19-36. They are 
included in both counts, thus the sum of the month 1-18 and 19-36 percentages can be larger than the totals in Exhibit 4-1. 


Statistical significance levels based on differences between research groups: *** 1 percent level; ** 5 percent level; * 10 percent level. 


4.2. Impact on Enrollment and Credits 


This section reports the impact of Pathways to Healthcare on enrollment in college and receipt 
of college credits three years after random assignment. In the short term, the program increased 
enrollment in college in all months following random assignment. As noted below, these impacts 
persisted. 


The earlier report did not address impacts on credits earned, because many Level 1 and Level 2 
programs resulted in noncredit “clock-hour” certificates. Clock hours are defined by the program 
and are the weekly number of hours a student spends attending class or other instructional 
activities (e.g., labs) that count toward completing a program of study. Each clock-hour program 
has a total number of clock hours required (e.g., Medical Office Clerk is 120 hours).°? Other 
PCC programs, including a number of Level 2 and Level 3 ones, are for credit. In credit 
programs, students need to complete a certain number of credits to successfully complete the 
program (e.g., Medical Assistant is 28 credits). Some programs offer clock-hour and credit 


°3 Neither the College Readiness class nor its lab had clock-hour requirements, and no credential was 


awarded for completion of the class or lab. 
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certificates (e.g., Nursing Assistant is 120 clock hours or four credits). The predominance of 
clock-hour programs in Pathways to Healthcare has implications for impacts on credits. LPN, 
one of the more common Level 3 programs, is clock hour. 


m Access to Pathways to Healthcare increased enrollment in college and full-time 
equivalent (FTE) enrollment. 


Exhibit 4-3 plots trends in enrollment in any college by quarter after random assignment; 
quarters 1 through 6 show the impact during the first follow-up period. The treatment group 
continued to enroll in college at significantly higher rates than the control group through quarter 
13. However, as summarized below, levels of enrollment never exceeded 40 percent of the 
treatment group and 30 percent of the control group. By quarter 17, about 9 percent of the 
treatment group and 10 percent of the control group was enrolled in training, with no detectable 
difference between the two. 


Exhibit 4-3: Impact on Any College Enrollment by Quarter, Three Years after Randomization 


45 


Treatment Grou 
+12.8*** = p 


40 $15.25 axe Control Group 


35 +11.6*"* 
30 +8.8*** 


25 


20 


Percent of Any College Enrollment (%) 


Q1 Q2 Q3 Q4 Q5 Q6 QZ OQ QI QIN QI1 Q1I2 Q13Z Q14 QI5 Q16 QI7 
Quarter Since Random Assignment 


Source: PCC records. 


Note: Enrollment estimates within each quarter are exploratory outcomes and statistical significance is based on two-tailed tests. Sample 
size is 609 in the treatment group and 608 in the control group. 


Statistical significance levels based on differences between research groups: *** 1 percent level; ** 5 percent level; * 10 percent level. 
Pathways to Healthcare also increased FTE enrollment in college, though the magnitude of the 


increase varies by data source. The pre-registered secondary outcome measures FTE 
enrollment using PCC records. According to those data, Pathways to Healthcare increased FTE 
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enrollment by 0.8 months, a relative impact of 19 percent (Exhibit 4-4). According to the three- 
year follow-up survey responses, Pathways to Healthcare increased FTE enrollment in college 
by 2.8 months, a relative impact of 56 percent (Exhibit 4-4). Although not registered as a 
secondary outcome, the measure based on survey records seems to have more validity—only 
with survey responses was the distribution of FTE months comparable to all students in the 
PCC records.” 


The larger survey estimate of FTE enrollment impact is more consistent with the larger impact 
on credentials reported in Section 4.1. There are at least two reasons the PCC records may 
underestimate FTE enrollment. First, students who enrolled in PCC Center for Training and 
Development (CTD) courses had very high rates of withdrawal (for which they were not counted 
as enrolled in PCC records even if they did participate for some time). Second, we made 
assumptions in order to “translate” CTD clock hours into attempted hours. For instance, we 
assumed that 45 CTD clock hours required the same level of effort as a three-credit PCC 
course. Excluding participants who withdrew from our enrollment calculations and our 
translating of CTD clock hours to FTE enrollment could have resulted in an underestimate of 
FTE enrollment from PCC records. 


Exhibit 4-4: Three-Year Impacts on Other Enrollment Measures 


Treatment Control Impact Standard _ Relative 
Outcome Group Group (Difference) Error Impact p-Value 

PCC Records 
Number of college credits (#) 8.55 6.27 +2,28** (0.83) +36.7% .003 
FTE (full-time-equivalent) months 
enrolled in college (#) 5.41 4.66 +0.75** (0.45) +16.1% .048 
Completed >10 FTE months enrolled in 
college (%) 18.2 17.3 +0.9 (2.1) +5.2% 661 

Sample size 609 608 
FTE months enrolled in college (#) 7.76 4.97 +2.79*** (0.53) +56.1% <.001 
FTE months enrolled in any school (#) 5.91 5.21 +0.70 (0.51) +13.4% .169 
Current enrollment in training or 
education (%) 15.5 12.2 +3.4 (2.1) +27.9% .109 

Sample size 530 510 


Source: PCC records and PACE three-year follow-up survey. 

Note: Secondary outcomes are bolded and statistical significance is based on one-tailed tests; exploratory outcomes are not bolded and 
Statistical significance is based on two-tailed tests. “Relative Impact” represents impacts as a percentage of the corresponding control group 
mean (i.e., 100 x [impact/control group mean). 

@ Outcome based on blend of PCC records and three-year follow-up survey data. 

Statistical significance levels based on differences between research groups: *** 1 percent level; ** 5 percent level; * 10 percent level. 


The impacts for FTE months enrolled and college credits are smaller than that for credentials. 
For reference, the relative impact for postsecondary credentials requiring a year or more of 


This is especially evident for students who enrolled in courses in the CTD. See Appendix E for more 


information. 
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college was 62 percent, and for any postsecondary credential it was 143 percent; but for college 
credits it was 37 percent and for FTE months enrolled it was 16 percent.°° The Pathways to 
Healthcare program’s rich advising services offer one plausible explanation for this difference— 
treatment group members had access to dedicated advisors who helped students navigate 
paperwork and other pre-course requirements (e.g., background checks). 


m Pathways to Healthcare increased receipt of college credits. 


In the three-year follow-up period, the treatment group earned an average of 8.6 credits—an 
increase of 2.3 credits over the control group. As noted above, however, most Pathways to 
Healthcare programs resulted in noncredit certificates. Thus, we did not expect to see a large 
accumulation of credits even when we documented an impact on credentials. 


°° The relative impact for FTE months is 56 percent using the three-year follow-up survey and 16 


percent using PCC records. Both are much smaller than the relative impact on credentials, though the 
survey-based estimate is closer. 
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The Pathways to Healthcare theory of change suggests that positive impacts on occupational 
training certifications will lead to higher levels of earnings and employment, particularly 
healthcare-related employment. We did not assess earnings and employment impacts for the 
short-term report at 18 months because we hypothesized that it would be too early for the 
impacts to emerge. However, it seems reasonable to expect impacts after three years because 
program participants would have had enough time to attain one or more credentials, including 
ones taking a year or more of college, and gain healthcare-related employment and earnings 
associated with their credentials. As described in Chapter 4, the program had impacts on 
postsecondary credential attainment, including credentials that take at least one year of college. 


This chapter reports whether training impacts translated into impacts on earnings, employment, 
and other measures of job quality three years after random assignment. The confirmatory 
outcome for the earnings and employment domain, or the outcome we use to determine 
whether Pathways to Healthcare is meeting its goals, is average quarterly earnings in follow-up 
quarters 12-13, which corresponds to months 37 through 42 after random assignment. 


5.1. Impact on Earnings 


We used NDNH wage records to determine whether earnings impacts emerged by the end of 
the three-year follow-up period. Exhibit 5-1 summarizes these findings. 


m Pathways to Healthcare had no detectable impact on average quarterly earnings in 
follow-up quarters 12-13. 


The top row in Exhibit 5-1 below shows that the difference in average quarterly earnings in 
quarters 12 and 13 between the treatment and control groups was positive but close to zero 
(+$17) and not statistically significant. 


As is true in all evaluations of job training programs, the impact was estimated with uncertainty. 
When we incorporate that uncertainty into a range of plausible impacts, we estimate that the 
true impact could be as large as +$356 or as small as -$322.°° Most of this range is either 
negative or not sufficiently larger than zero to be meaningful.°’ 


The increase in college enrollment reported in Chapter 4 combined with the lack of impacts on 
earnings suggests that the treatment group substituted work for training or reduced their hours 


°° These values are the endpoints for a 90 percent confidence interval for average earnings in quarters 


12 to 13. 


7 The upper end of this plausible range is not large relative to results from some recent studies. For 


instance, about two years after random assignment, impact for those assigned to the Wisconsin 
Regional Training Partnership (WRTP) in the Sectoral Employment Impact Study (SEIS) was $782 
per quarter (Maguire et al. 2010) and for Per Scholas (one provider in the WorkAdvance 
Demonstration) was $937 per quarter (Hendra et al. 2016). 
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while in school. If training improves earnings as implied by the theory of change, we should 
observe improvements after training is completed and participants return to work or increase 
their hours. In Chapter 3 we reported that occupations associated with Level 1 trainings had 
lower average annual earnings than occupations associated with Level 2 or 3 trainings. The lack 
of earnings impacts might reflect that more treatment group members completed a Level 1 
course than a higher step on the pathway. As a result, over the entire period from randomization 
to quarter 13, the treatment group had earned about $2,800 Jess than the control group.°® 


Exhibit 5-1: Three-Year Impacts on Earnings 


Treatment Control Impact Standard _ Relative 
Outcome Group Group (Difference) Error Impact p-Value 

Confirmatory Outcome: Average 4,175 4,158 +17 206 +0.4% 467 
quarterly earnings Q12-Q13 ($) 
Total Earnings ($) 

In last year of follow-up (Q10-Q13) 15,648 16,284 -636 733 -3.9% .386 

Since randomization (Q1-Q13) 40,107 42,921 -2,814* 1,681 -3.9%* .094 

Sample size 609 608 


Source: National Directory of New Hires. 


Note: Confirmatory and secondary outcomes are bolded and statistical significance is based on one-tailed tests; exploratory outcomes are 
not bolded and statistical significance is based on two-tailed tests. “Relative Impact” represents impacts in column 3 as a percentage of the 
corresponding control group mean (i.e., 100 x [impact/control group mean)). 


Statistical significance levels based on differences between research groups: *** 1 percent level; ** 5 percent level; * 10 percent level. 


Exhibit 5.2 reports the trend in earnings by quarter from before random assignment to the 18th 
follow-up quarter, which is the last quarter for which we have earnings information for the full 
sample.°? In the two quarters before random assignment (Q-2 and Q-1 in the graph), earnings 
fell for both the treatment and control groups.®° Following random assignment, average 
quarterly earnings begin to grow but there is no detectable difference between the treatment 
and control groups. In each of quarters 6 through 11, the control group had higher earnings than 
the treatment group. These quarters presumably correspond to the quarters during which the 
treatment group remained in training but the control group transitioned to the labor market. The 
negative impacts reflect lost wages for the treatment group during training. In quarters 12 
through 18 there is no difference between the treatment and control groups in quarterly 
earnings. 


°° Our estimates of cumulative earnings do not discount or control for inflation. Discounting would 


amplify lost earnings during the early quarters after random assignment and reduce any gains in later 
quarters, making the net difference even larger. Controlling for inflation is not crucial because inflation 
over the study period was approximately 2 percent per year (Bureau of Labor Statistics 2019). As 
well, inflation affects the treatment and control groups equally, and our goal here is to estimate 
whether there has been any increase in earnings following training, rather than to precisely estimate 
the magnitude of those earnings. 

°° The 18th follow-up quarter corresponds to 54 months (or 4.5 years) after random assignment. 


®° This is known as the “Ashenfelter dip” (Ashenfelter 1978). 
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Exhibit 5-2: Impact on Average Earnings in Successive Follow-up Quarters 
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Source: National Directory of New Hires. 
Note: Earnings estimates within each quarter are exploratory outcomes and statistical significance is based on two-tailed tests. Sample size 


is 609 in the treatment group and 608 in the control group. 
Statistical significance levels based on differences between research groups: *** 1 percent level; ** 5 percent level; * 10 percent level. 


5.2 Impact on Employment 


This section examines impacts on the level of employment and job characteristics as captured 
in the three-year follow-up survey. These offer additional information on the earnings estimates 
reported above. 


m Pathways to Healthcare had no detectable impact on employment as of three 
years after random assignment. 
Exhibit 5-3 shows that the treatment and control groups had similar employment levels at the 
time of the three-year follow-up survey, which is consistent with analysis of administrative 
earnings data. Slightly less than two thirds of both the treatment and control groups reported 
employment at the time of survey follow-up, and we do not detect any difference between them. 
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Exhibit 5-3: Three-Year Impacts on Employment and Career Progress 


Treatment Control Impact Standard _ Relative 
Outcome Group Group (Difference) Error Impact p-Value 

Employed at survey follow-up (%) 63.9 62.3 +17 (3.0) 2.7% 291 
Employed and: (%) 

Earning $14 per hour or more@ 21.2 24.4 -3.2 (2.6) -13.1% 887 

Paid under $10 per hour 10.1 10.4 -0.3 (1.9) -2.9% 872 

Paid $10 to $13.99 per hour 31.7 26.8 +4.9* (2.9) 18.3% .088 

Working in the healthcare field 

(self-classification) 39.4 34.8 +4.6* (3.0) 13.2% .066 

Working in a healthcare occupation 

(any industry) 20.7 21.4 -0.7 (2.7) -3.3% 803 

Working in a job requiring at least 

mid-level skills’ 22.0 21.0 +1.0 (2.6) 4.8% 349 


Indicators of Job Quality 


Employed and: (%) 


Working at least 32 hours per week 44.8 42.6 +2.2 (3.2) 5.2% .500 

Working straight day, evening, or 

night shifts 50.9 51.2 -0.3 (3.3) -0.6% 928 

Working in job that offers health 

insurance 43.5 42.9 +0.7 (3.1) 1.6% 833 

Working in job with supportive 

working environment¢ 31.3 31.1 +0.2 (3.0) 0.6% .942 
Sample size 530 510 


Source: PACE three-year follow-up survey. 

Note: Secondary outcomes are bolded and statistical significance is based on one-tailed tests; exploratory outcomes are not bolded and 
Statistical significance is based on two-tailed tests. “Relative Impact” represents impacts in column 3 as a percentage of the corresponding 
control group mean (i.e., 100 x [impact/control group mean]). 

4 $14 per hour is the 60" percentile of the wage distribution for control group members who were employed at survey follow-up. 

> O*NET Job Zone 3 or higher. 


© A job is considered to have a supportive working environment if the worker reports a rich combination of family-friendly policies, helpful 
coworkers and supervisors, high job satisfaction, generous fringe benefits, and opportunities for advancement. 


Statistical significance levels based on differences between research groups: *** 1 percent level; ** 5 percent level; * 10 percent level. 


Exhibit 5-4 shows the time trend in quarterly employment using NDNH records. The time trend 
shows very little difference in employment over time (the difference in quarter 6 is statistically 
significant at the 10 percent level).°* 


To supplement the quarterly earnings reported in the previous section, we tested whether the 
program could generate a moderate initial boost in wages with the expectation for further impact 
over time (a key assumption in the theory of change). We define these jobs using the 60th 


= Though the impacts are consistent, the levels of employment differ slightly between the three-year 


survey (Exhibit 5-3) and the administrative records (Exhibit 5-4). Survey-reported levels of 
employment are less than 65 percent, but administrative records indicate levels of employment 
around 70 percent. This difference is likely because the survey reflects employment at a single point 
in time whereas the administrative records reflect employment at any time during each quarter. 
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percentile of the wage distribution for control group members who were employed, which 
corresponds to about $14 per hour (Exhibit 5-3). Less than one quarter of both the treatment 
and control groups were employed at jobs that paid $14 per hour or more at the time of the 
survey, and we do not detect evidence of a program impact. 


Exhibit 5-4: Employment Impacts by Quarter 
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Source: National Directory of New Hires. 


Note: Employment estimates within each quarter are exploratory outcomes and statistical significance is based on two-tailed tests. Sample 
size Is 609 in the treatment group and 608 in the control group. 


Statistical significance levels based on differences between research groups: *** 1 percent level; ** 5 percent level; * 10 percent level. 


m Pathways to Healthcare increased self-reported employment in the healthcare 
field. 


The three-year follow-up survey asked study participants about their employment status, and for 
those working at the time of the survey, the characteristics of their job. As Exhibit 5-3 above 
shows, nearly 40 percent of the treatment group self-reported employment in the healthcare 
field, an increase of 5 percentage points over the control group. 


The survey also included three open-ended questions about the kind of work done, usual 
activities completed, and the job title. We converted these into a U.S. Department of Labor 
Standard Occupational Classification (SOC) code and used that code to classify employment in 
the healthcare sector (See Appendix C). Using this measure there is no detectable impact on 
employment in a healthcare occupation. 
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One explanation for this finding is that more treatment group members are working in the 
healthcare field but in occupations that the SOC system does not classify as being healthcare 
jobs (e.g., an IT-related job at a hospital).°* Because impacts for these two outcomes do not 
agree and we suspect some of the healthcare jobs PCC trained for were classified in the SOC 
system as non-healthcare employment, we place more weight on the pre-registered secondary 
outcome (self-reported employment in the healthcare field). 


m Pathways to Healthcare had no detectable impact on measures of job quality. 


The bottom panel of Exhibit 5-3 above reports treatment and control group members’ 
assessment of job quality. Treatment group members were no more likely than control group 
members to report that their current job required “at least mid-level skills,” classified as jobs with 
O*NET Job Zone 3 or higher.® About one in five study participants in both the treatment and 
control groups reported employment in this type of job. Similarly, the program did not improve 
other measures of job quality, such as working in a job for at least 32 hours per week or working 
straight shifts. Treatment group members were no more likely to report working in a job that 
offers health insurance, or working in a job with a supportive working environment. 


= To explore this hypothesis further, we tabulated the occupations that were self-reported as in the 


healthcare field but were not coded in the SOC system as healthcare jobs. The most common of 
these occupations was Personal Care Aide. In the 2010 SOC system Personal Care Aide was not 
considered a healthcare job. This status was revised in 2018. 


88 O*NET defines occupations in Job Zone 3 as those that “need medium preparation.” Most 


occupations in this zone require training in vocational schools, related on-the-job experience, or an 
associate degree. O*NET lists Medical Assistant as an example of an occupation in Job Zone 3 
(O*NET 2019). 
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6. Impacts on Other Life Outcomes 


This chapter examines whether Pathways to Healthcare affected other life outcomes, including 
those related to career knowledge and support, family economic well-being, parental 
engagement, and child outcomes. The program’s theory of change implies that outcomes in 
these domains will improve as a result of increases in education and training that lead to more 
favorable earnings and employment outcomes. As discussed in Chapter 4, treatment group 
members were more likely to earn a range of postsecondary credentials. However, training thus 
far has not translated into career-track jobs with higher earnings, as seen in Chapter 5. Asa 
result, the grounds for more distal effects are somewhat uncertain. 


6.1 Impact on Career Knowledge, Availability of Career Supports, and 
Psycho-social Skills 


This section reports Pathways to Healthcare’s impacts on career knowledge, availability of 
career supports, and psycho-social skills. Improvements to these outcomes are hypothesized to 
boost postsecondary educational attainment and career progress (Judkins et al. 2018). 


m The Pathways to Healthcare program increased access to career supports, but not 
confidence in career knowledge. 


Three years after random assignment the treatment group reported greater access to career 
supports (an effect size of 0.15), but we do not detect a difference in confidence in career 
knowledge (Exhibit 6-1 below).® These findings are consistent with the short-term report at 18 
months. 


4 An effect size of 0.15 corresponds to approximately 56 percent of the treatment group scoring higher 


than the comparison group (Judkins et al. 2008). 
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Exhibit 6-1: Impacts on Career Knowledge, Career Supports, and Psycho-social Skills 


Treatment Control Impact Standard Effect 
Outcome Group Group (Difference) Error Size p-Value 
Confidence in career knowledge? 3.35 3.34 +0.01 (0.04) +0.02 385 
| fl ot o- ojele 
Access to career supports’ 1.59 1.54 +0.05** (0.02) +0.15** 011 
Perceived career progress¢ 3.10 2.93 +0.17*** —— (0.06) +0,19*** .003 
Grité 3.34 3.37 -0.04 (0.03) -0.07 .303 
Core self-evaluation® 3.43 3.44 -0.01 (0.03) -0.01 838 
Index of life challenges! 1.69 1.65 +0.04 (0.04) +0.07 .250 
Perceived stress9 2.01 1.97 +0.04 (0.05) +0.05 427 
Social support" 3.67 3.69 -0.02 (0.03) -0.05 436 
Sample size 530 510 


Source: PACE three-year follow-up survey. 

Note: Secondary outcomes are bolded and statistical significance is based on one-tailed tests; exploratory outcomes are not bolded and 
Statistical significance is based on two-tailed tests. “Effect Size” represents impacts in column 3 as a fraction of the pooled standard deviation 
across the treatment and control groups. See Appendix C for a description of outcome measures. 

@Seven-item scale tapping self-assessed career knowledge; response categories range from 1=strongly disagree to 4=strongly agree. 

> Six-item scale tapping self-assessed access to career supports; response categories range from 1=no to 2=yes. 

© Three-item scale on whether reaching long-range education goals and employment goals and whether on career path; response categories 
range from 1=strongly disagree to 4=strongly agree. 

4 Eight-item scale measuring self-assessed persistence and determination; response categories range from 1=strongly disagree to 4=strongly 
agree. 

® Twelve-item scale measuring self-assessed self-efficacy; response categories range from 1=strongly disagree to 4=strongly agree. 

‘ Five-item scale of situations that could interfere with school, work, job search, or family responsibilities. 

9 Four-item scale measuring self-reported perceived stress; response categories range from 1=never to 4=very often. 

 Ten-item scale measuring availability of social support; response categories range from 1=strongly disagree to 4=strongly agree. 

Statistical significance levels based on differences between research groups: *** 1 percent level; ** 5 percent level; * 10 percent level. 


Among other outcomes in this domain, the treatment group reported greater perceived career 
progress (effect size of 0.19, another finding that remains unchanged from the short-term 
report).®° We do not detect differences in psycho-social skills, including grit, core self-evaluation, 
life challenges, perceived stress, and social support. 


6.2. Impact on Family Economic Well-Being 


This section reports impacts for several measures of family economic well-being, including 
health insurance coverage, receipt of means-tested benefits, unsecured debt and student loan 
debt, and financial status. 


°° The measure of perceived career progress is a three-item measure that combines progress toward 


longer-term educational goals, progress toward longer-term employment goals, and a self-report that 
one is on a career path. In a sensitivity analysis we removed the education component of the 
measure to test whether impacts were being driven by that component. Removing that component did 
not change the results. 
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The Pathways to Healthcare theory of change suggested that a number of program 
components, including academic and non-academic advisors, employment workshops, and 
financial assistance to pay for training and related expenses, would lead to positive outcomes 
on a range of family economic well-being measures, including receipt of means-tested public 
benefits, debt levels, and signs of financial distress. The expected direction of some effects is 
less clear at the three-year mark. For example, non-academic advising could facilitate 
enrollment in a means-tested program such as TANF or Medicaid to make it easier for students 
to persist in college. Career-track employment and higher earnings could reduce the need for 
these benefits, but not all treatment group members are employed in career-track jobs (see 
Chapter 5). Access to financial supports would lead to lower student debt. However, students 
may have had to take out loans or borrow from their families to pay for non-academic expenses 
while they were in school (e.g., rent) leading to higher levels of debt. 


@ Pathways to Healthcare had no detectable impact on most measures of family 
economic well-being. 


As Exhibit 6-2 below shows, nearly 90 percent of both the treatment and control groups had 
health insurance coverage from any source three years after random assignment, with no 
detectable difference between the two. And though access to Pathways to Healthcare did not 
reduce receipt of means-tested public benefits overall, more of the treatment group (33 percent) 
received Medicaid than the control group (26 percent). 


We posit several explanations for why access to Pathways to Healthcare might increase receipt 
of Medicaid. First, Medicaid expansion in Arizona in 2013 coincided with this study’s follow-up; 
the support services offered to those in Pathways to Healthcare may have encouraged 
application for Medicaid and/or removed some of the institutional barriers that might have 
inhibited application. Another possibility is that treatment group members had more access to 
Medicaid via a stronger connection to healthcare training and employment, which could have 
reduced the administrative and informational barriers associated with Medicaid enrollment. Still 
another could be that treatment group members had a greater understanding of the importance 
of health insurance and made sure they were covered in any way they could (employer- 
sponsored or Medicaid). 
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Exhibit 6-2: Impacts on Varied Measures of Family Economic Well-being 


Treatment Control Impact Standard Relative 
Outcome Group Group (Difference) Error Impact p-Value 
Has health insurance coverage (%)@ 88.6 89.5 -0.9 (2.0) -1.0% .667 
[Receiptofmeans-tested benefits 
Any means-tested public benefits (%) 60.3 52.9 +7.4 (3.1) 14.0% 991 
TANF 4.4 2.9 +15 (1.2) 51.7% .203 
SNAP 31.6 30.4 +13 (2.7) 4.3% 641 
Medicaid 32.6 25.6 +6.9* (2.8) 27.0% .013 


Average student debt amount ($) 


Participant's student debt 1,995 2,054 -59 (388) -2.9% 439 

Parental student debt 89 54 +35 (56) 64.7% 532 
Unsecured debt of $5,000 or more (%) 33.3 29.6 +3.7 (2.9) 12.5% .212 
Financial Status 
Any signs of financial distress¢ (%) 56.2 53.7 +2.5 (3.1) 4.7% .789 
Average monthly household income ($) 2,638 2,766 -128 (114) -46% 261 
Average monthly personal income ($) 1,444 1,426 +18 (68) 1.3% 794 
Didn't experience food insecurity (%) 89.6 90.8 “1.2 (1.9) -1.3% 041 


Sample size 530 510 


Source: PACE three-year follow-up survey. 


Note: Secondary outcomes are bolded and statistical significance is based on one-tailed tests; exploratory outcomes are not bolded and 
Statistical significance is based on two-tailed tests. “Relative Impact” represents impacts in column 3 as a fraction of the corresponding 
control group mean (i.e., 100 x [impact/control group mean]). 

2 Has health insurance coverage includes the offer of healthcare by an employer or actual receipt of health insurance if not offered by an 
employer. 

> Unsecured debt is debt other than student debt and secured debt (mortgages and title loans). Spousal debt included. 

© Signs of financial distress is a flag for utility disconnects, delayed health/dental care, hunger, or trouble paying bills or making ends meet. 
Statistical significance levels based on differences between research groups: *** 1 percent level; ** 5 percent level; * 10 percent level. 


Similarly, Pathways to Healthcare did not have an impact on debt or signs of financial distress 
overall. There was no detectable impact on either student debt or parental student debt. More 
than half of both the treatment and control groups reported signs of financial distress overall, but 
there was no difference between the two. There were also no differences between the treatment 
and control groups in household income, personal income, or adequacy of food for the 
household. 


6.3. Impact on Parental Engagement and Child Outcomes 


This section reports impacts for several outcomes related to parental engagement and child 
well-being for study participants with children under age 18 at the time of random assignment. 
The Pathways to Healthcare program provided no direct services to children, but it is plausible 
that effects might flow from parents’ experiences with the program and increases in their 
educational attainment, employment, or income. It is possible that parents who pursue training 
in a field, complete the training, and move into employment in their field feel they accomplished 
a life goal, which could lead them to encourage their children to do well in school and attend 
postsecondary education—a positive for children. As well, it is possible that parents could role- 
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model the value of education, for example, by doing homework together with their children— 
also a positive. Conversely, it is also possible that parents who are at school or working have 
less ability to engage with and supervise their children—a negative for children. 


Note that the three-year follow-up survey asked these questions only of parents who had 
children under age 18 at the time of random assignment, and for some questions, only parents 
of children in grades K-12.°° Because not all sample members were parents and the parents 
sample size is smaller than the overall sample size, the analysis is not well powered to detect 
small differences in impacts.° 


m Pathways to Healthcare had no detectable impact on parental engagement and 
child outcomes. 


The top panel of Exhibit 6-3 reports impacts for parental engagement with all children under age 
18. More than three quarters of both treatment and control group members believe their children 
will graduate college. Less than a quarter of each group classified themselves as highly 
engaged parents. Finally, parents in both the treatment and control groups reported similar 
levels of self-efficacy for helping their child navigate school. 


The bottom panel of Exhibit 6-3 above reflects parental report of child outcomes for children in 
kindergarten through grade 12. As with parental engagement, we do not detect any differences 
in child outcomes between the treatment and control groups. About 10 percent of both groups 
reported their child had ever repeated any grades (including prior to random assignment). On 
average, children of parents in both the treatment and control groups were late for school about 
one day and were absent about 1.4 days in the prior month. 


°° The sampling frame for parents with children included those whose projected interview date for the 


three-year survey came before the end of the summer in the year their child turned age 18. Those 
projections were not always correct. As a result, some of the children in the sampling frame were 
older than age 18 at the time of the three-year survey. 


8? Our analysis plan (Judkins et al. 2018) noted that we would report impacts for all-grade and K-12 


questions in programs where we had least 200 children. 
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Exhibit 6-3: Impacts on Parental Engagement and Child Outcomes 


Treatment Control Impact Standard _ Relative 
Outcome Group Group (Difference) Error Impact p-Value 

All Children 
Parent believes child will graduate 
college (%) 76.7 83.9 -7.2 (5.2) -8.6% 169 
Highly engaged parent (parent almost 
always present for meals and other daily 
family activities) (%) 20.8 24.7 -3.8 (5.3) -15.4% 472 
Parent self-efficacy for helping child 
navigate school@ 3.48 3.42 +0.1 (0.07) +0.10 436 

Sample size 162 120 
Child repeated any grades (%) 10.0 8.5 +15 (3.9) 17.6% 104 
Days child late for school last month (#) 1.05 1.08 -0.03 (0.26) -2.8% 915 
Days child absent from school last month 
(#) 1.39 1.41 -0.01 (0.28) -0.7% .957 

Sample size 139 99 


Source: PACE three-year follow-up survey. 


Note: Statistical significance is based on two-tailed tests. “Relative Impact” represents impacts in column 3 as a fraction of the corresponding 
control group mean (i.e., 100 x [impact/control group mean]). For the scale variable (parent self-efficacy), we report effect size rather than 
relative impact. “Effect Size” represents impacts in column 3 as a fraction of the pooled standard deviation across the treatment and control 


groups. 


2 Parental self-efficacy based on seven items (e.g., “l know how to help my child in school”) rated from 1=disagree very strongly to 6=agree 
very strongly. See Appendix C for more details on child outcome measures. 
Statistical significance levels based on differences between research groups: *** 1 percent level; ** 5 percent level; * 10 percent level. 
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PCC received an HPOG 1.0 grant award to implement Pathways to Healthcare. The program— 
which operated between 2010 and 2016—defined and sought to guide participants in five 
occupational training pathways along with providing financial, academic, and other supports. 
The program aimed to increase students’ skills and receipt of credentials of value to employers. 
This report documents program impacts on postsecondary training, earnings and employment, 
and other life outcomes three years after random assignment (with follow-up extended to up to 
five years after random assignment where administrative records are available). 


The program’s theory of change posits that impacts on education outcomes will translate into 
career-track employment and earnings impacts. Evidence on the two confirmatory outcomes in 
this study, however, is mixed: 


e Pathways to Healthcare did increase receipt of a credential requiring a year or more of 
college to earn by 7 percentage points—the confirmatory outcome in the education 
domain. 


e The program had no detectable impact on the confirmatory outcome in the earnings and 
employment domain—increase in average quarterly earnings in follow-up quarters 12- 
13. Our estimated impact of +$17 per quarter is not significantly different from zero. The 
uncertainty associated with the estimate implies a plausible range of -$322 to +$356. 
This range implies PCC did not have a meaningful impact on earnings in the first three 
years after random assignment. Furthermore, estimated earnings impacts remained 
small and statistically insignificant when we extended the analysis to nearly five years. 


Beyond these two confirmatory outcomes, Pathways to Healthcare increased short-duration 
educational credentials, employment in the healthcare field, access to career supports, and 
perceived career progress. However, the program did not generate detectable impacts on other 
measures of interest, including job quality, family economic well-being, or psycho-social skills. 


This concluding chapter provides possible explanations for the Pathways to Healthcare impact 
findings. It then describes implications of the findings for other job training programs. It ends 
with questions for future research. 


7.1 Possible Explanations for Pathways to Healthcare Impact Findings 


This section explores three possible explanations for the impact findings: (1) a large share of 
the treatment group did not engage in training; (2) few participants advanced beyond their first 
step (Level 1) on the pathway; and (3) the impact on long-duration credentials (those taking a 
year or more of college) within three years was not large enough to generate earnings impacts. 
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e A large share of treatment group members did not engage in the program. 


As the theory of change indicates, non-enrollment in any training affects impacts on short-term 
credentials, subsequent credentials, and earnings. One potential explanation for the absence of 
statistically significant earnings impacts at the three-year follow-up is that 44 percent of 
treatment group members had not engaged in any type of occupational training during this time. 
As the short-term (18-month) report indicated, there are many possible reasons for non- 
engagement, including difficulty combining work, school, and family responsibilities. We would 
not expect impacts to emerge for earnings or other outcomes for this large share of treatment 
group members who received no training from Pathways to Healthcare. This has repercussions 
for the estimated average impact of the program. Because non-enrollees are included in impact 
estimates and are not expected to have any impact from the program, they dilute the average 
impact for the overall treatment group. 


e The program mostly increased short-duration credentials with modest potential to 
lift wages. 


Pathways to Healthcare implemented five healthcare pathways, most starting at Level 1 and 
continuing to Level 3. The impacts on postsecondary attainment reported in Chapter 4 indicate 
much larger impacts on receipt of short-duration credentials (requiring less than a year of 
college to earn) than on receipt of long-duration ones (requiring at least a year): a 20 
percentage point impact for short-duration credentials compared to a 7 percentage point impact 
for long-duration credentials. Those short-duration credentials are typically associated with 
Level 1 programs, and administrative records indicate that the majority of treatment group 
members did not move beyond Level 1.°° As noted in Chapter 1, average wages associated 
with Level 1 occupations do not differ substantially from those that could be earned outside of 
the healthcare field (e.g., personal care, food preparation). A short-duration credential may 
position the worker on the initial step of a career ladder, but we would not expect such a 
credential to generate a meaningful impact on earnings at this time point. Unless the large share 
of the treatment group who earned Level 1 credentials move along their pathways to higher- 
paying jobs through additional training, it is unlikely that they will earn more than those who did 
not complete any training. This will be a question of interest for the long-term follow-up study 
currently underway. 


These results are consistent with the three-year findings for the HPOG 1.0 Impact Study (Peck 
et al. 2019). That study pooled data from 23 grantees (including PCC). Comparing Pathways to 
Healthcare to the HPOG 1.0 Impact Study findings is akin to comparing it to the average 


8 As noted in Chapter 2, the study used an intent to treat approach, which captured impacts for all 


sample members, regardless of whether those assigned to the treatment group actually received any 
services. 


°° For instance, within the 18-month follow-up period, the most common short-duration credential was 


Certified Nursing Assistant. This first step on the Nursing pathway is a five-week program (nine 
weeks if it includes Level 2 Patient Care Technician training). Less common Level 1 programs were 
Phlebotomy (six months), the first step on the Medical and Physician Support pathway; and Medical 
office Clerk/Specialist (one month), the first step on the Medical Office pathway. 
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HPOG 1.0 program. That study found that, as with Pathways to Healthcare, the most popular 
trainings among HPOG 1.0 programs were short-duration trainings, such as Nurses’ Aide. That 
study also found that impacts on earnings for HPOG 1.0 grantee programs were similar to those 
of Pathways to Healthcare—no impact on average quarterly earnings during quarters 12 and 13 
after random assignment. 


e Impacts on long-duration credentials are too small to generate earnings impacts. 


Chapter 1 noted that occupations associated with higher-level credentials have higher average 
hourly wages. As reported, wages for some Level 2 and 3 occupations (e.g., Surgical 
Technician, Licensed Practical Nurse) are higher than for others (e.g., Medical Assistant, 
Pharmacy Technician). More treatment group enrollment in courses to prepare for these higher- 
wage occupations appears to be needed in order to detect impacts on earnings. 


All Level 2 and 3 credentials required at least one year of full-time college. The extent to which 
Pathways to Healthcare increased receipt of these credentials and created opportunities to 
obtain these higher-wage jobs has implications for the impact we expect to emerge for quarterly 
earnings. The relatively small impact on receipt of long-duration credentials—7 percentage 
points by the three-year follow-up—implies that we should expect a similarly small impact on 
earnings. Rough calculations suggest that in order to detect an impact on earnings, the impact 
on postsecondary credentials requiring at least a year of college would need to be considerably 
larger than 7 percentage points. 


’° To demonstrate, consider the following calculation. Suppose wages are the same for employed 


treatment group and control group members, both for those who earn no credential and for those who 
earn a short-duration credential. We would not expect an impact on earnings for this part of the 
sample, all else equal. If all treatment group and control group members who earned longer-duration 
credentials were employed in higher-wage occupations at the three-year follow-up, we would expect 
an impact of approximately +$250 to +$350 in quarterly earnings. To increase this to +$500 in 
quarterly earnings, the impact on long-duration credentials would need to be roughly 10 to 13 
percentage points. 


For demonstration, we derive this +$250 to +$350 as follows. Per Chapter 1, those employed in 
higher-wage occupations can expect an increase in average wages of approximately $8 to $10 per 
hour relative to the wages they could earn with a short-duration credential; an improvement of 7 
percentage points implies an increase in average wages of approximately 50 to 70 cents per hour 

(7 percent x $8-$10 per hour). Assuming roughly 500 hours of work per quarter, we would expect an 
earnings impact of only approximately +$250 to +$350 in quarterly earnings. Given that not all Level 2 
and Level 3 occupations have higher wages, this is an upper bound for the likely impact. 
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7.2 Implications for Programs 


The mixture of positive impacts in the education domain and no detectable impact in the 
employment domain suggests the following implications for Pathways to Healthcare and similar 
sectoral job training programs. 


e Programs can have a positive impact on credential attainment for non-traditional 
students. 


Pathways to Healthcare and similar programs targeted low-income, low-skilled adults, many of 
whom had not been in an educational setting in years. The impact on any credential shows that 
non-traditional students can engage in, persist, and complete programs. The entry-level (Level 
1) trainings in four different pathways ensured there were diverse options for participants 
starting on the initial pathway rung. 


e Programs with multi-step pathways should encourage participants to seek higher- 
level credentials. 


Pathways to Healthcare requested that program completers work for at least six months prior to 
starting a follow-on training to ensure that the occupation was a good fit.’* After such a gap, it 
might be useful for program staff to more aggressively reach out to participants and encourage 
them to return for higher-level trainings. Additionally, if participants are not working in their field 
of training because they did not like the work or could not find a job, such outreach is an 
opportunity to encourage participants to try a different type of program (e.g., medical records 
rather than direct patient care). 


Programs may also need to help participants prepare for higher-level trainings. Most of the 
Pathways to Healthcare Level 2 and 3 programs had considerably higher math and reading 
level requirements than Level 1 programs. To address this, PCC helped participants prepare for 
the LPN program assessment test by implementing a pre-LPN reading group. The instructor 
focused on test-taking skills and reading strategies while the group read a few pages of the 
assigned book. A memoir about an author's stroke, The Diving Bell and the Butterfly, was 
selected because of its extensive health-related vocabulary. 


e Programs should identify and address causes of nonparticipation. 


There are several reasons why individuals might not start a training program. Non-traditional 
students may have difficulty combining work and school and may not be able to afford to stop 
working in order to pursue their education full time. Additionally, a delay between enrolling ina 
program and starting classes, either because of extensive pre-training requirements (e.g., 
paperwork, assessments) or infrequent course start dates, can negatively affect participation. 
As described in the 18-month report, the average time between random assignment and the 
start of Pathways to Healthcare training was almost four months. This delay may have 
increased the fraction of the treatment group who did not participate in training. More attention 
to reducing this delay seems worthwhile. Toward the end of the grant period, PCC implemented 


™ The one exception was the CNA to Patient Care Technician pairing. 
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a bi-monthly group orientation to engage participants between random assignment and training, 
and describe the steps required prior to entering training so that participants would have a full 
understanding of the program timeline. 


e College-based programs should strengthen employment services to help connect 
training completers find credential-related employment. 


Bridging the gap between program completion and employment can be difficult for programs. In 
particular, college programs may have limited experience providing employment services 
beyond program-related internships and clinical placements. PCC partnered with Pima County 
One Stop to provide employment services. The program also funded an employment specialist 
to work with employers in the community and advise the program on hiring trends. Over the 
course of the program, PCC and PCOS developed new job search-related workshops and 
launched a student-alumni networking group. However, according to the 18-month survey, less 
than one-quarter of treatment group members reported receipt of employment services. 


In preparing its application for its second HPOG grant, PCC staff conducted an employer survey 
and met with members of PCC Career Advisory Councils. Employers and Advisory Council 
members helped select the pathways and provided information about the academic knowledge 
program completers would need to meet employer demands. Employers also pointed to the 
importance of participants obtaining experiential learning prior to starting a job. In response, 
PCC and PCOS planned to leverage employer relationships for in-program experiential learning 
experiences (e.g., clinical experiences, internships, apprenticeships) and hiring program 
completers. 


7.3 Open Questions 


Three years after random assignment, Pathways to Healthcare improved educational progress, 
specifically receipt of credentials, but did not improve earnings. A future report will focus on the 
impact of Pathways to Healthcare on earnings approximately six years after random 
assignment. This section highlights questions for that longer-term research. 


e Will treatment group members continue to progress in college faster than control 
group members? 


More than 18 percent of treatment group members earned a postsecondary credential taking a 
year or more of college to earn. As noted above, most of these credentials are associated with 
higher-paying jobs. A key consideration is whether more treatment group members will continue 
to earn long-duration credentials relative to control group members. Some were still in training 
at the end of the three-year follow-up period (16 percent of the treatment group; 12 percent of 


”?  Itis possible that participants received more services than they reported if the response categories 


did not resonate with them. For example, discussions about career options were embedded in many 
Pathways to Healthcare services, including advising and employment services; however, as “career 
counseling” was not a distinct program component, students may not have reported receiving it (and 
the same could be true for control group members). No other data are available to corroborate these 
estimates. 
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the control group) and may still complete a Level 2 or 3 credential. Others could return for 
additional training, particularly if they can continue to access financial assistance and support 
through HOPES. Our analyses from Chapter 3 found some evidence of treatment group 
members returning for additional training through HOPES to a greater degree than control group 
members did. 


A key question is whether continued progress in college by the treatment group will differ from 
the control group’s progress. As of this report, low levels of enrollment for both the treatment 
and control groups seem inconsistent with substantial additional progress in college and in 
particular with differentially more progress for the treatment group than for the control group. 


It is, however, possible that the treatment group may make more progress in college relative to 
the control group. For instance, though we know the treatment and control groups remain 
enrolled at similar rates, we do not know from the college records what types of trainings or 
credentials the control group are pursuing. We cannot rule out meaningful differences in these 
trainings or credentials. In addition, the program’s emphasis on pathways may have instilled an 
interest in returning for additional training and an understanding of the next pathway step among 
the treatment group that differs from the control group’s interest and understanding. If this were 
the case, we might observe differences in college enrollment at a later date despite the fact that 
enrollment rates do not differ as of this three-year follow-up. 


e Will more members of the study sample find employment in healthcare? 


Consistent with its goals, the Pathways to Healthcare program increased self-reported 
employment in the healthcare field. However, the levels of employment in healthcare for both 
the treatment group and control group were lower than anticipated—roughly one third of the 
study sample reported employment in the healthcare field. For reference, Peck et al. (2019) 
reported more than 50 percent of the HPOG 1.0 treatment group was employed in healthcare 
after three years. 


There are several reasons why more Pathways to Healthcare treatment group members were 
not working in the healthcare field at the point of follow-up. One possibility is the low rate of 
engagement with occupational training. Another is that levels of overall employment are lower 
for Pathways to Healthcare than for the broader HPOG 1.0 treatment group. Also, as noted 
above, wages associated with entry-level credentials in healthcare may not be higher than what 
workers could earn in another, perhaps less difficult, job. Furthermore, some participants may 
have received a credential but not attempted or passed a licensing exam. As a result, these 
participants would be unable to work in the healthcare field for which they trained. Whether 
more members of the study sample will find employment in healthcare in the future remains an 
open question. 


e Will impacts on earnings emerge over a longer follow-up period? 


As noted above, earnings gains are associated with higher-level credentials, and they can take 
time to emerge. Treatment group members earned these credentials at higher rates than control 
group members did. Nevertheless, neither the levels nor the impacts are large. It is unlikely 
Pathways to Healthcare will have earnings impacts unless a much larger share of treatment 
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group members enroll in and complete long-duration credentials, including Level 2 and 3 
credentials and associate degrees. As of five years after random assignment, there were some 
impacts on long-duration credentials, including associate degrees. However, it is unclear 
whether this impact will increase enough in the future to produce detectable impacts on 
earnings. 


e Will impacts differ for the next generation of career pathways programs? 


Pathways to Healthcare was an early career pathways program. The PCC staff who designed 
the program did not have a large base of evidence from which to draw. Rather, program staff 
blended program components the literature suggested were important to helping the target 
population persist in and complete training. As more rigorous evidence has become available, 
career pathways programs have adjusted their approaches. For instance, the Funding 
Opportunity Announcement (FOA) for HPOG 2.0 was much more explicit in requiring applicants 
to structure their programs in line with the career pathways framework, including placing 
stronger emphasis on participants returning for additional training and supporting them in long- 
term training.’ Future evaluations of career pathways programs, such as the National 
Evaluation of HPOG 2.0, should consider the extent to which those programs have improved 
implementation of career pathways programs and the extent to which improved implementation 
leads to better training and labor market outcomes. 


73 The HPOG 2.0 FOA required applicants to provide a narrative description or diagram that 


demonstrated “how all of the proposed healthcare occupational trainings and basic skills training are 
linked together in one or more career ladders and/or lattices, with priority given to occupations that 
are expected to be full time, have regular hours, offer benefits, and/or have strong potential for 
advancement.” The FOA notes that by themselves, many entry-level healthcare occupations do not 
have these characteristics, “which is one of the reasons why a career ladder and/or lattice can be so 
critical to a participant’s success.” Applicants had to include the SOC code for each proposed 
occupational training, the average starting wage in the grantee’s service area, and how many 
students will start in each training (HHS/ACF/OFA 2015). This added language expanded upon the 
characteristics of training activities described in the HPOG 1.0 FOA, which required programs to 
“support participants’ advancement along a defined career pathway, such as an articulated career 
ladder, if such a pathway exists in the healthcare industry, or that involve developing such pathways 
where they do not currently exist” (HHS/ACF/OFA 2010). 
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